Dealing with RF is tricky part of electronic design, even though I never had significant issues with GSM in my prototypes. Advancing in WLS project however I discovered a nasty issue with micro SD prone to lock up on interference from GSM built-in antenna, even located in significant distance. To my surprise half of different tested brands suffered from it. Thru extra tests, selection and SD replacement action I was able to greatly improve stability of long-running system.
First hand-made board had issues with ground loop between main and GSM board leading to MCU resets. On redesign and shrinking PCBs for factory production this issue had gone. I had couple months of flawless run on balcony test station and one field station. Over time second station was deployed as well as significant changes to firmware were made. I started to see missing data points caused by spurious resets. Since troubles started few months back I was almost sure I introduced some bugs in firmware. In the field station did reset irregularly: from tens of hours to tens of days, once a week on average. The same hardware and firmware however run on my desk for 2 weeks like a charm. Also balcony test station never did a single reset. Yet field stations started living their own lifes. Conspiracy theories started buzzing in my head: probe cable too long? Weak DC/DC blocks? Another subtle defect in my code or some library? Frustration was enormous. I have sieved out some minor defects and added extra countermeasures like hardware watchdog and RTOS level software watchdog sniffing for stalled tasks. Extra tracking data led to nowhere as stations still experienced occasional unpredictible hiccup. Even worse they hung sometimes for several hours before reset leaving no clue in log files.
Even coaxial cable touching PBC below microSD traces leads to lockup
When one station stopped responding for two days I decided to replace the whole box with new one and to take dismantled one without stopping it and debug on my workbench. What I have seen was that MCU/RTOS was still running fine while any I/O operation on SD generated errors I was printing out to debug UART. Since my implementation supports hot replacement of SD card, every time it detects write error it tries to remount file system. And each time I have seen f_mount(pFs, “”, 1): 0x01 on my console, the error code representing low level I/O problem. In other words SPI driver implementation was unable to talk to SD card. I have seen it running for long minutes without a chance to reinitialize SD card. Even with extra power cycle added to code (with 500ms delay between off and on again) it did not help either and station locked up again on another occasion.
I recalled that roughly year ago I have witnessed SD card lock up: GSM antenna was almost touching SD card and once disconnection from SD happened. I was amazed how strong was electromagnetic wave emitted by antenna so that spikes induced on 2-3cm PCB traces choked SD completely. That time I thought that antenna inside the case is located far enough (about 15cm away) to lower chance for such misbehavior down to zero. I tried to replicate this misbehavior and failed trying all possible antenna figures around SD card.
Excerpt from debug UART when SD works properly:
2015-07-15 20:19:38 DBG gsm GPRS context attached. 2015-07-15 20:19:42 DBG gsm GPRS link established. 2015-07-15 20:19:43 INFO gsm IP address assigned: 10.203.232.25 2015-07-15 20:19:43 DBG gsm Listing store files. 2015-07-15 20:19:43 DBG gsm Store listed 4 file(s) in batch of 10. 2015-07-15 20:19:49 INFO gsm Sending batch of 4 file(s), 3404 bytes. 2015-07-15 20:19:49 DBG gsm Pushing file #1 (15201842.NTF). 2015-07-15 20:19:50 DBG gsm Pushing file #2 (15201914.NTF). 2015-07-15 20:19:50 DBG gsm Pushing file #3 (15201915.LOG). 2015-07-15 20:19:55 DBG gsm Pushing file #4 (15201937.NTF). 2015-07-15 20:20:00 INFO gsm Data successfully submitted. 2015-07-15 20:20:00 DBG gsm Removing batch of files. 2015-07-15 20:20:00 DBG gsm Listing store files. 2015-07-15 20:20:00 DBG gsm Store listed 0 file(s) in batch of 10. 2015-07-15 20:20:02 INFO gsm Sending data finished. |
Excerpt from debug UART on failing SD card; lines below “Sending batch of…” are not saved:
2015-07-15 20:16:59 DBG gsm GPRS context attached. 2015-07-15 20:17:03 DBG gsm GPRS link established. 2015-07-15 20:17:04 INFO gsm IP address assigned: 10.204.213.2 2015-07-15 20:17:04 DBG gsm Listing store files. 2015-07-15 20:17:04 DBG gsm Store listed 7 file(s) in batch of 10. 2015-07-15 20:17:10 INFO gsm Sending batch of 7 file(s), 5213 bytes. f_mount(NULL, "", 1): 0x00 f_mount(pFs, "", 1): 0x01 2015-07-15 20:17:10 DBG gsm Pushing file #1 (15201521.NTF). 2015-07-15 20:17:12 ERR gsm Cannot open 15201521.NTF, error 1. f_mount(NULL, "", 1): 0x00 f_mount(pFs, "", 1): 0x01 2015-07-15 20:17:12 DBG gsm Pushing file #2 (15201542.NTF). 2015-07-15 20:17:13 ERR gsm Cannot open 15201542.NTF, error 1. f_mount(NULL, "", 1): 0x00 f_mount(pFs, "", 1): 0x01 2015-07-15 20:17:14 DBG gsm Pushing file #3 (15201543.LOG). f_mount(NULL, "", 1): 0x00 f_mount(pFs, "", 1): 0x01 2015-07-15 20:17:16 ERR gsm Cannot open 15201543.LOG, error 1. f_mount(NULL, "", 1): 0x00 f_mount(pFs, "", 1): 0x01 |
Days were passing by as each test required long run to see the issue. I was clueless. Until a day all of a sudden while taking a shower (so typical) it clicked in my head: my balcony test station has never replaced heavily used no-name 1GB microSD while deployed stations were given brand new Samsung 1GB cards. Also my workbench PCB had no-name disposable SD card mounted. Could that be that cheapish pre-owned cards were way more resistive to EMI than same density new and popular microSD? I rushed to my desk just to run the board again and again with different SD cards each time manipulating GSM antenna near the SD slot. Results were devastating: five Samsung 1GB cards, five SanDisk 2GB cards and one no-name card I tested were hanging up instantly every time as soon as I moved antenna or its coaxial cable near SD. In contrast another five no-name 1GB cards from single batch and one Kingston 1GB cards never locked up.
3:2 for failing cards
Metal box to force stronger antenna signal
Since cellular signal in the city is strong (80-90% of maximum reception) antenna interference is weak as weak signal emission is enough. To simulate low reception condition I used metal box to create Faraday’s cage; in that condition signal dropped down to 40% and now even with antenna placed away from SD slot, Samsung and SanDisk cards were locking instantly on data transmission. Drastic measure led to drastic side-effects: series of intense interference tests killed two SD cards: they became read-only with frozen content on it and any attempt to write to or format card simply failed, no matter operating system and computer architecture.
EDIT: First week after SD cards replacement lapsed and over that period only one reset in 1 of 4 stations occured while reporting unusually week (<30%) signal reception and strong antenna emission. Previously the same period of time would lead to 3-7 resets across all stations. Sounds like extra RF shielding around PCBs will be necessary to completely reduce interference. Experience by experiments motto applied again.
See also other related articles:
[posts_by_tag tag=”WLS”]
Hello,
what about metal shielded micro SD sockets ? This should solve the problem in an elegant and effective way and they do not cost significantly more then regular plastic version.
Best regards !
PS. I’ve just discovered your blog and I think I’ll be a frequent visitor here 😉
Mark, I already have metal socket for microSD, it is roughly 50% transparent though and it is obviously not enough. I also see that some signal lines between MCU and SD can be source of trouble. My goal was to fix current design, not redesigning it, minimizing changes needed to apply to devices mounted in the field.
PS. You are welcome with appreciation for your flattering post scriptum; I try to write only when I see some value in message to be delivered 🙂