 Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me. |
Forum Index : Microcontroller and PC projects : Software testing
| Author | Message | ||||
| zeitfest Guru  Joined: 31/07/2019 Location: AustraliaPosts: 492 |
Is there some consensus on software testing standards, ie on software for micro's ? |
||||
| Volhout Guru Joined: 05/03/2018 Location: NetherlandsPosts: 4343 |
Hi zeitfest, Forum member thwill (Tom) has created a test "suite" to validate consistency and correct implementation of commands. I think it was created for CMM2. For early picomite I it was expanded and run. But Tom is very busy releasing MMB4L 0.7 so he won't have any focus for picomite now. And recently so many commands where added and changed for picomite, the test suite would first need an update, maybe even an overhaul. I think the test suite is not designed to test correctness of the implemented commands, but detect degradation/differences between successive builds. Volhout Edited 2024-12-20 17:11 by Volhout PicomiteVGA PETSCII ROBOTS |
||||
thwill Guru Joined: 16/09/2019 Location: United KingdomPosts: 4066 |
I'm not even sure what question the OP was really asking ? As @Volhout said I have written a tool called sptest which can be used to write unit-tests in MMBasic using a rough approximation of the xUnit test pattern. This was designed for the Colour Maximite 2, MMBasic for Windows and MMB4L. To use it on the PicoMite you have to flatten the #INCLUDE structure using CMM2 RUN or some sort of transpilation. Amongst other things I use this tool to support an incomplete set of tests of the MMBasic firmware functionality: https://github.com/thwill1000/mmb4l/tree/main/tests. Unfortunately I have not had the opportunity to run these on recent PicoMite builds; I've been having some problems with my tooling for transferring code to the PicoMite and haven't been able to make the time to resolve it or use an alternative. Best wishes, Tom Edited 2024-12-20 20:16 by thwill Game*Mite, CMM2 Welcome Tape, Creaky old text adventures |
||||
disco4now Guru Joined: 18/12/2014 Location: AustraliaPosts: 916 |
I make extensive use of a slightly modified version of Toms sptest framework. I have run the tests on Picomites, MMX eXtreme, Armmite F4 and Armmite H7. It is particularly useful when porting the latest Picomite features to the other platforms. As errors are reported and fixed, if possible I modify the tests so they would pickup that issue and use this to see if it exists in the other platforms. I have an addon to MMEdit to manage the include files at load time. Below is the output from the Armmite H7. The tests range in complexity from just seeing if the command exists, seeing if it does what is expected, seeing what is says if you give it incorrect parameters, seeing the correct pins become reserved and some test hardware functionality e.g. looping TX and RX on com ports allows testing of various speeds, parity etc. > *sptest --all --verbose Building list of tests ... Added... tst_crypt.bas Added... tst_array.bas Added... tst_pins.bas Added... tst_bits.bas Added... tst_math.bas Added... tst_string.bas Added... tst_longstring.bas Added... tst_error_handling.bas Added... tst_memory.bas Executing tests ... tst_crypt.bas --all --verbose test_md5_fmt: PASS (1/1) test_md5_given_string: PASS (28/28) test_md5_given_long_string: PASS (27/27) test_md5_file: PASS (18/18) test_xxtea_block_encrypt: PASS (21/21) test_xxtea_block_decrypt: PASS (21/21) Execution time: 0.773557 s PASS (6/6) tst_crypt.bas --all --verbose --base=1 test_md5_fmt: PASS (1/1) test_md5_given_string: PASS (28/28) test_md5_given_long_string: PASS (27/27) test_md5_file: PASS (18/18) test_xxtea_block_encrypt: PASS (21/21) test_xxtea_block_decrypt: PASS (21/21) Execution time: 0.774242 s PASS (6/6) tst_array.bas --all --verbose test_capacity: PASS (1/1) test_copy: PASS (40/40) test_fill: PASS (10/10) test_case_sens_bsearch: PASS (41/41) test_case_insens_bsearch: PASS (36/36) test_join_floats: PASS (25/25) test_join_ints: PASS (25/25) test_join_strings: PASS (39/39) Execution time: 0.092658 s PASS (8/8) tst_array.bas --all --verbose --base=1 test_capacity: PASS (1/1) test_copy: PASS (40/40) test_fill: PASS (10/10) test_case_sens_bsearch: PASS (41/41) test_case_insens_bsearch: PASS (36/36) test_join_floats: PASS (25/25) test_join_ints: PASS (25/25) test_join_strings: PASS (39/39) Execution time: 0.092442 s PASS (8/8) tst_pins.bas --all --verbose test_call_fn_with_wrong_type: PASS (1/1) test_gosub_label_not_found: PASS (1/1) test_adcstart: NO ASSERTIONS test_adcpins: PASS (336/336) test_comports: FAIL (7/37) 25: Assert equals failed, expected "9600,S2,OC<13><10>@COM1" but actually "timeout@COM1" 26: Assert equals failed, expected "9600,S2,OC<13><10>@COM2" but actually "timeout@COM2" 27: Assert equals failed, expected "9600,S2,OC<13><10>@COM3" but actually "timeout@COM3" 28: Assert equals failed, expected "9600,S2,OC<13><10>@COM4" but actually "timeout@COM4" 29: Assert equals failed, expected "DE TEST COM2<13><10>@COM2" but actually "timeout@COM2" 32: Assert equals failed, expected "DE TEST COM2<13><10>@COM2" but actually "timeout@COM2" 36: Assert equals failed, expected "COM2vsPWM2D<13><10>@COM2" but actually "timeout@COM2" test_pwmpins: PASS (16/16) test_i2cpins: PASS (24/24) test_spipins: PASS (36/36) test_countpins: PASS (48/48) test_irpin: PASS (2/2) test_filesize: PASS (9/9) Execution time: -185.453241 s FAIL (1/11) tst_bits.bas --all --verbose test_set: PASS (1/1) test_clear: PASS (1/1) test_get: PASS (136/136) Execution time: 0.02059 s PASS (3/3) tst_bits.bas --all --verbose --base=1 test_set: PASS (1/1) test_clear: PASS (1/1) test_get: PASS (136/136) Execution time: 0.021495 s PASS (3/3) tst_math.bas --all --verbose test_base conversions: PASS (3/3) test_angle: PASS (9/9) test_math_crc8,crc12,crc16,crc32: PASS (6/6) test_pseudo_rnd: PASS (21/21) test_cosh: PASS (1/1) test_sinh: PASS (2/2) test_tanh: PASS (1/1) test_log10: PASS (1/1) test_atn,atan2,atan3: PASS (10/10) test_sd: PASS (1/1) test_chi: PASS (3/3) test_set: PASS (1/1) test_mean: PASS (1/1) test_add: PASS (1/1) test_sum: PASS (1/1) test_scale: PASS (1/1) test_shift: PASS (3/3) test_window: PASS (8/8) test_crossing: PASS (3/3) test_array maths ,add,sub,mult,div: PASS (4/4) test_complex numbers: PASS (23/23) test_max: PASS (1/1) test_min: PASS (1/1) test_fft_given_sine_wave: PASS (1/1) test_fft_inverse,magnitude,phase: PASS (9/9) test_correl: PASS (2/2) test_interpolate: PASS (1/1) test_insert: PASS (1024/1024) test_slice: PASS (1/1) test_median: PASS (1/1) test_vectors: PASS (3/3) test_matrix: PASS (5/5) test_determinant: PASS (3/3) test_quaterions: PASS (6/6) test_sort integer array: PASS (2/2) Execution time: 1.665283 s PASS (35/35) tst_math.bas --all --verbose --base=1 test_base conversions: PASS (3/3) test_angle: PASS (9/9) test_math_crc8,crc12,crc16,crc32: PASS (6/6) test_pseudo_rnd: PASS (20/20) test_cosh: PASS (1/1) test_sinh: PASS (2/2) test_tanh: PASS (1/1) test_log10: PASS (1/1) test_atn,atan2,atan3: PASS (10/10) test_sd: PASS (1/1) test_chi: PASS (3/3) test_set: PASS (1/1) test_mean: PASS (1/1) test_add: PASS (1/1) test_sum: PASS (1/1) test_scale: PASS (1/1) test_shift: PASS (3/3) test_window: PASS (8/8) test_crossing: PASS (3/3) test_array maths ,add,sub,mult,div: PASS (4/4) test_complex numbers: PASS (23/23) test_max: PASS (1/1) test_min: PASS (1/1) test_fft_given_sine_wave: PASS (1/1) test_fft_inverse,magnitude,phase: PASS (9/9) test_correl: PASS (2/2) test_interpolate: PASS (1/1) test_insert: PASS (1024/1024) test_slice: PASS (1/1) test_median: PASS (1/1) test_vectors: PASS (3/3) test_matrix: PASS (5/5) test_determinant: PASS (3/3) test_quaterions: PASS (6/6) test_sort integer array: PASS (2/2) Execution time: 1.719302 s PASS (35/35) tst_string.bas --all --verbose test_centre: PASS (3/3) test_equals_ignore_case: PASS (4/4) test_lpad: PASS (2/2) test_is_plain_ascii: PASS (128/128) test_next_token: PASS (57/57) test_next_token_given_quotes: PASS (23/23) test_quote: PASS (9/9) test_replace: PASS (5/5) test_rpad: PASS (2/2) test_trim: PASS (12/12) test_unquote: PASS (6/6) test_sort string array: PASS (40/40) test special chars with OPTION ESCAPE: PASS (16/16) test special chars in DATA strings: PASS (2/2) test special case &00 and 000: PASS (8/8) Execution time: 0.126345 s PASS (15/15) tst_string.bas --all --verbose --base=1 test_centre: PASS (3/3) test_equals_ignore_case: PASS (4/4) test_lpad: PASS (2/2) test_is_plain_ascii: PASS (128/128) test_next_token: PASS (57/57) test_next_token_given_quotes: PASS (23/23) test_quote: PASS (9/9) test_replace: PASS (5/5) test_rpad: PASS (2/2) test_trim: PASS (12/12) test_unquote: PASS (6/6) test_sort string array: PASS (40/40) test special chars with OPTION ESCAPE: PASS (16/16) test special chars in DATA strings: PASS (2/2) test special case &00 and 000: PASS (8/8) Execution time: 0.126748 s PASS (15/15) tst_longstring.bas --all --verbose test_append: PASS (13/13) test_clear: PASS (1/1) test_copy: PASS (1/1) test_concat: PASS (4/4) test_lcase: PASS (1/1) test_lcompare: PASS (3/3) test_left: PASS (2/2) test_lgetbyte: PASS (12/12) test_lgetstr: PASS (2/2) test_linstr: PASS (3/3) test_llen: PASS (1/1) test_load: PASS (1/1) test_mid: PASS (1/1) test_replace: PASS (1/1) test_resize: PASS (4/4) test_right: PASS (1/1) test_setbyte: PASS (1/1) test_trim: PASS (2/2) test_ucase: PASS (1/1) test_print: PASS (2/2) Execution time: 0.068055 s PASS (20/20) tst_longstring.bas --all --verbose --base=1 test_append: PASS (13/13) test_clear: PASS (1/1) test_copy: PASS (1/1) test_concat: PASS (4/4) test_lcase: PASS (1/1) test_lcompare: PASS (3/3) test_left: PASS (2/2) test_lgetbyte: PASS (12/12) test_lgetstr: PASS (2/2) test_linstr: PASS (3/3) test_llen: PASS (1/1) test_load: PASS (1/1) test_mid: PASS (1/1) test_replace: PASS (1/1) test_resize: PASS (4/4) test_right: PASS (1/1) test_setbyte: PASS (1/1) test_trim: PASS (2/2) test_ucase: PASS (1/1) test_print: PASS (2/2) Execution time: 0.071675 s PASS (20/20) tst_error_handling.bas --all --verbose test_function_wrongtype: PASS (1/1) test_gosub_nolabel: PASS (1/1) test_error_normal: PASS (5/5) test_error_in_interrupt: PASS (5/5) test_error_given_pipe: PASS (2/2) test_interrupt_does_not_swallow_skip: PASS (3/3) test_on_error_skip_2: PASS (3/3) test_interrupt_does_not_ignore: PASS (2/2) test_error_skip_with_if_block: PASS (2/2) Execution time: 2.412484 s PASS (9/9) tst_error_handling.bas --all --verbose --base=1 test_function_wrongtype: PASS (1/1) test_gosub_nolabel: PASS (1/1) test_error_normal: PASS (5/5) test_error_in_interrupt: PASS (5/5) test_error_given_pipe: PASS (2/2) test_interrupt_does_not_swallow_skip: PASS (3/3) test_on_error_skip_2: PASS (3/3) test_interrupt_does_not_ignore: PASS (2/2) test_error_skip_with_if_block: PASS (2/2) Execution time: 2.413518 s PASS (9/9) tst_memory.bas --all --verbose test_set: PASS (256/256) test_set_byte: PASS (256/256) test_set_short: PASS (130/130) test_set_word: PASS (66/66) test_set_integer: PASS (34/34) test_set_float: PASS (34/34) test_copy: PASS (3/3) test_copy_byte: NO ASSERTIONS test_copy_short: NO ASSERTIONS test_copy_word: NO ASSERTIONS test_copy_integer: PASS (5/5) test_copy_float: PASS (5/5) test_copy_given_overlap: PASS (16/16) test_pack,unpack: PASS (1/1) test_print,input: PASS (3/3) Execution time: 0.126566 s PASS (15/15) tst_memory.bas --all --verbose --base=1 test_set: PASS (256/256) test_set_byte: PASS (256/256) test_set_short: PASS (130/130) test_set_word: PASS (66/66) test_set_integer: PASS (34/34) test_set_float: PASS (34/34) test_copy: PASS (3/3) test_copy_byte: NO ASSERTIONS test_copy_short: NO ASSERTIONS test_copy_word: NO ASSERTIONS test_copy_integer: PASS (5/5) test_copy_float: PASS (5/5) test_copy_given_overlap: PASS (16/16) test_pack,unpack: PASS (1/1) test_print,input: PASS (3/3) Execution time: 0.12722 s PASS (15/15) Total execution time: 35 s Latest F4 Latest H7 |
||||
| zeitfest Guru Joined: 31/07/2019 Location: AustraliaPosts: 492 |
Thanks It is a general question more than anything specific. I have written test scripts for testing my own interpreter but probably should use a standard package of some sort. I have little faith in testing standards at the best of times but they seem to be a required formality. I worked for a UK-headquartered software company in decades past, which stated that its software was accredited to ticketed [name withheld] standards per ISO yada yada yada. The backend database server used was typically DEC RDB v5. A new client had bought the package and I went out to install it and give the user training course, carefully scheduled by the client to avoid interrupting production in the plant. RDB was by then v6 and the company line was that the software worked fine with it and was quality accredited to the highest standards etc. It would not install. Contacted UK support. So, rescheduled for the next week while a slightly updated version X arrived. Next week : It would not install. Contacted UK support. Got a condescending message back explaining that training in simple installation would be useful. So, rescheduled for the next week. Tried exhaustively to install. Next week : It would not install. Contacted UK support. Got half a fax back, which started a reply - then the fax page stopped halfway. After a pause there was a new message explaining that it would not install under RDB v6. If v5 was installed then upgraded to v6, which the developers had done, the package would then install OK.. Unfortunately RDB v5 was discontinued, Compaq was taking over DEC and was iffy about continuing DEC products anyway, and the upgrade path was not viable. So had to explain to client that the product they had bought would not, after rescheduling three times, install. An interesting conversation in front of the half-dozen new users sitting there ready for their weeks training course .. So much for the product testing, accreditation, multiple signatures, etc. Luckily there was a different version that used another data server, so ( after rescheduling again ) when the client shelled out a serious additional amount, that installed ok. In Aus, information systems used for pharmaceutical systems/manufacturing etc have to be validated to a standard where each detailed operation is inspected and signed off. So they are seriously more expensive. For non-health systems the standard is mainly "trust the software vendor"  . Then you get the situation, where the functional spec is written after the product is created...magically the quality checking finds exact conformance ! Who would have thought ?! Probably the preferred path is to regard a test protocol to be created and included as part of programming each function. Probably not practical though.  |
||||
 Print this page |
