Display freeze issue

[r2r]

We are having some issues with display freezing on the field. Something very similar to https://forum.4dsystems.com.au/node/45880 but not as consistent as that post there. We don't see the display blank out or reset but mostly the display freezes and no communication between the display and our main motherboard. We are using GEN4-ULCD-32DCT-CLB mounted on an aluminum enclosure. The enclosure has good EMC shielding and the motherboard has EMI/RF protection as well. The display is mounted onto the enclosure using the supplied adhesive.

We update the display about at a rate of 10 Hz from our motherboard and sometimes the customer complains about the display freezes while everything else works with our motherboard. The only way out is to power cycle our whole system. This is not ideal for them with their industrial operation. This is an industrial production environment with high current and AC high voltage devices around our device. We think this could be something related to static but don't know for sure. The problem does not occur as often and as consistent as the post that we referred to before but it does occur every so often (once a month). However, with a 24/7 operation that is still not acceptable for our customer to power cycle our device since this can upset their production process.

Any ideas?

If static is a problem how can we mitigate it? Our mother board seems to just work fine but just the display freezes. We think this is more like the display is not in sync with the motherboard anymore. Is there a way in which we can safely discharge the static? How would we do that with a cover glass display?

[James_4DSystems]

Hello,

Sorry to hear you are having issues like this.

So you are using a gen4-uLCD-32DCT-CLB. What Environment are you using in Workshop4? Designer, ViSi, Genie, Serial, and are you using any PRO features or all the standard Workshop4 features?
What is your Host?
What is your baud rate?
Are you using a custom protocol, or is it Serial or Genie comms?
How long are your cables between Host and Display?
Is there any indication of events leading up to the freeze, as to what might be causing it?
Is the Display 100% locked up? Do you have any buttons on the screen that when you touch them they animate and depress, such as a Winbutton? If you do, and they still animate, then this can indicate the display is still running OK, but the comms between the display and the host have gone out of Sync, so it then comes down to what environment in Workshop4 you are using as to what solution may be possible.

Lets start with that

Regards

[r2r]

Hello James,

- Visi-Genie. I don't think we have any PRO features on this particular version.
- Host is a custom board communicating with 200000 baud. We have our own library adapted from the 4D system library. The cable is the standard flex cable that comes with the display.
- We are not aware of any indication to events leading to the freeze since it is our customer who reports the problem. We can find out.
- Unfortunately we are not able to replicate the problem but we are assuming that this is a complete freeze. What if it is just a sync problem? How could we tackle it?

Thanks,

[r2r]

Actually looking at the library (adapted from GenieArduino) there is a _genieLinkCount which would call the _genieResync() function to resync. So I am not too sure if that might be a resync. We don't reset the display through the reset line after a timeout.

[James_4DSystems]

We have our own library adapted from the 4D system library.

Which library did you use to base yours on? The mainstream genieArduino one? or did you use the BETA version?
The BETA version has some techniques in there to help recover from out of sync situations.
If your comms get a random byte in the stream for some reason, maybe external noise injection etc, then it can lead to an out of sync situation which it may not be able to self recover. The Beta library had a much better chance of recovering.
Let me discuss internally and ill come back with hopefully some more information

[r2r]

The main library was adapted in 2016 and I don't think we kept with the latest branches or BETA version. So I'm going to guess it is the mainstream genieArduino.

[tonton81]

Hello r2r,

I'm understanding you are having difficulties with the freezing issue, but if possible to check when it freezes if touching a button on the screen still outputs the display's TX wire? If so, your mcu is out of sync with the library. This can be of one of 3 issues.

1) bad bytes entered the stream due to interference, perhaps cabling capacitance or "ringing" effect of signals.

2) the baudrate precision. Understand that many different microcontrollers on the market have a percentage (%) of error rate on certain baud rates. The 4D documents specify the exact rate at those specified speeds. 115200 may not essentially be 115200 on some mcus. It may be 114800 (this is an example). It is better to match the exact rates as per the documentation of both the display and the mcu in order to have a good percentage rate in order to avoid issues with corrupted data in stream. You may find perhaps 115200 works better than 2000000, depending on the precision of your microcontroller, most hardware engineers design around 115200, so you may be safe there, but all other speeds, you must account for error rate fluctuations.

3) Interrupts. Check to see you don't sit in an interrupt routine too long, you may lose serial data. Unfortunately, this would be a coding effort of your own project and no library would be able to fix this.

The production library has no reassuring recovery of above 3 points, and a beta library was made in an attempt to handle this situation. However, the interrupt issue is the worst problem as we can't fix that, it's the core code in your runtime project.

I also like to mention that if you get TX tested while it's frozen, if your scope sees data it's a sync issue.
However! If there is a complete display lockup and you dont even see buttons being pushed when pushing them, I would recommend to change to a different SD brand, or pick up industrial grade one on the website. Not all SD cards are designed around SPI protocol, and manufacturers do not list them so easily to public.
Bad/incompatible SD cards will definately cause freezing issues. There is NO recovery for this you just have to change the card.

Tony

[r2r]

We greatly appreciate your input. Unfortunately we are unable to debug since the product is in the field at a customer location. Like I said this does not happen often. It has been reported by one customer who is running a 24/7 operation and happens about one or two times a month. And that too with one device on a particular line while the other devices are working just fine.

I have asked them to let us know if the buttons are still working, that is they still depress and change state. I am not too optimistic that we get the correct information since the information that we get would be at best third-hand information.

1. We think the bad byte is the main culprit. How can this be avoided? Cables connected to our device are shielded and the shield is connected to ground on one end to avoid any ground loops. Our host board has static ESD protection with TVDS but how can we properly discharge static from the display. Seems like the display is completely isolated with its cover glass and plastic housing. The only connection between the display and the host is that flexible flat ribbon cable. I am assuming the display has some TVDS or some other mechanism to dissipate static discharge to the GND. The operators working around this display may be charged since the manufacturing operation is around plastics. Any thing that we can do in the hardware to prevent the interference or the bad byte?

2. We will try and look at baud rate precision as well. But since we have no problems with other customers with many displays on the filed, we are reluctant to make such a significant change without a good reason. Our initial testing revealed that 200000 was the best baud for the host and the display. Do you have any doubt that the 200000 baud is not optimal for the display? We would suspect that if the baud rate was a problem then we should see it in-house and more often. Albeit, we will look into it as well.

3. We don't believe the interrupt is an issue since as mentioned before we have no problem with other similar devices on the same plant or the many other devices on the field.

Unfortunately we cannot scope the TX line when the display is frozen.

The SD card is an interesting one. We have seen some issues in the past when we didn't have an industrial SD card. With those issues the objects on the display would appear corrupted (after a while) and it was not possible to get the display working properly even after power cycles. The only way to recover the display was to use a new SD card. We have been using industrial SD card for a while now and we thought the issue was behind us. But now this is something we might also focus on. In one of the other threads I saw some mention about the 'read disturb' and we have Sandisk 8GB Industrial MLC MicroSD SDHC UHS-I Class 10 SDSDQAF3-008G with 'manual/auto read refresh' which is similar to 'read disturb.' Now the question is do you have some ideas on how to test if this card is good enough? I saw somewhere that some tests were done with multiple read cycles to the same block to get the problem simulated, how can this process be accelerated with a test? Does the 'manual/auto read refresh' functionality good enough? Do we need to buy the Phison cards? Also does it matter that we reformat the card to 4GB using the 4D systems reformat utility? Since our problem is infrequent compared to the number of displays we have in the field how can we know if the SD card is the problem?

Thanks,
Engineering

[James_4DSystems]

The read disturb testing can be done with a test program we have written, but its not a fast test. You ideally need to leave it running for 3 months and observe what it shows on the display. Read Disturb is not something that happens quickly in general, but there are some cards out there which do fail rather rapidly. I would imagine your Sandisk Industrial cards should be OK, however if you are wanting me to send you the program by email, just let me know. I can use your forum email address if that is suitable for you.

Regarding the baud rate, as Tony stated it does depend on the Host, however if you have bench tested it and its running for months, then I dont think this should be the issue. What processor is in your Host, just out of interest?

Regarding the bad byte, as mentioned already out BETA library does handle this better than the standard one, so you will have to look over that and see the changes and if your own library can be improved or not. Using your own library limits us somewhat in the help we can provide, as we know nothing about it other than it was based on our library.

One thing you could potentially incorporate into your library though is to query the display periodically. If you get no response from the display, have your host pause the sending of genie comms, and just send 1 byte at a time to the display until you get a NAK, then you know you are in sync with the packet, and start sending regular data again.

I hope this helps somewhat.

Regards

[r2r]

Oh okay. Yes, we might as well cross all the t's and dot all the i's. If you wouldn't mind sending to the e-mail address we will have a chance to check.

Our custom board has a Microchip dsPIC33EP512 running about about 70 MHz.

We will look at the BETA library to see what we can adopt. I think we do have some features that detect the presence of a display and not send any commands when the display is not detected. I thought we did have a way to recover but seems like it is not implemented. I don't think we send data one byte at a time. That is a good suggestion to get the display to sync back again with the host. We will definitely try that.

Thanks for all your suggestions.

[r2r]

We made some fixes to our code to reduce the issue with bad byte and re-syncing. But we are still hearing problems. We also got the feedback that some touch events are working. That is we can see the press event (with the change in icon) but that event does not have an effect on the host. Since the product is in the field we are unable to say if the display actually sends the transmit event.

Additionally we have a heart beat LED on the home screen that changes color at the rate of 1 Hz. And that is working. Since we send that info only when we are on the home screen we know that the host and the display are in sync. We are pretty convinced that it is not the issue with Baud Rate, or Interrupts. Since the display is receiving the information from the host.

The two possibilities are that our host is not processing the transmit event from the display. Or the display is not sending the transmit event. We know that our host functions even without the display and other UART events (Ethernet, RS485 and others are working just fine). Any suggestions?

The other commonality is that this happens when the display is on over night or has been working for several hours. Obviously a power cycle works but that is not acceptable for us.

Is it possible that the display freeze may be attributed to a static discharge from an operator in industrial environment? Usually we see this issue more often when the air is dry. This might be a different issue but surely coincidental.

[John Kevin]

Hi,

Thank you for the explanation. Can you confirm if these issues are the same as what you mentioned in your previous posts? with the same display?

It has been reported by one customer who is running a 24/7 operation and happens about one or two times a month. And that too with one device on a particular line while the other devices are working just fine.

Thank you and have a good day!

Best Regards,
Kevin

[r2r]

The issue is slightly different albeit the display freezes. However, I have personally noticed this issue in our lab and on our demo units.

We also came across this post: https://forum.4dsystems.com.au/node/68534 and would like to know if this could be a similar issue with newer hardware. Is there a known issue with latest hardware that we need to be aware of? How can be ground the display properly?

[John Kevin]

Hi,

Regarding the referred forum thread. The cause of the issue is the capacitor (C76) which is part of the Reset circuit for HW rev 1.5. The capacitor is there to assist with the suppression of ESD which can result in the processor getting reset, however, it also affects the displays start-up time. Removing the component resolves the issue.

Looking at your previous message, it seems that you are using gen4-uLCD-32DCT-CLB. The customer on that thread uses larger displays and is different from what you have.
The gen4-uLCD-32DCT-CLB still uses the HW rev 1.2 and there is no hardware revision involved.

However, I have personally noticed this issue in our lab and on our demo units.

Do you have the unit with you? Could you possibly emulate the issue and post here when does it occur?

Thank you and have a good day!

Best Regards,
Kevin