Announcement

Collapse
No announcement yet.

recovery after error

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • recovery after error

    Ok, so I have an uOLED-160 that's powered by a 800mAh LiPo through a boost converter. This is a low-power, battery operated device with occasional use, so I have an accelerometer-based watchdog system that enables the boost converter when the device is moved (i.e., the user sees no power switch). When in battery-powered mode, the thing works flawlessly. Eventually, though, the battery is depleted and needs to be charged.

    As the thing comes back to life, despite there being *more than enough* power from the charger to charge the display *and* run the video module simultaneously, it fails to boot cleanly. The symptom is always that the uSD drive is dysfunctional. No number of calls to initialize it will ever cause it to work.

    I guess my question is, how can I trigger a deep-enough reset of the module that the uSD *really* gets tried again? The init command is simply not doing the trick. What I'd really like to do is get the module to (I guess) go through the restart sequence again (or do *anything* that will get me out of the condition where my system refuses to run because of a zombie uSD device).

    This does not happen every time through this condition, so I think I'd be fine if I could simply kick the device a few times until it gets past that point in the code. Sometimes a twiddle of the reset line will do the trick, but I really don't want my users to be exposed to that form of input to the device (and I don't have an open I/O pin to twiddle it that way).

    Is there a way? Power will be present to the device, so if there's a command to power it down that will result in a restart when/if power is present, that might do the trick.

    Thanks for any suggestions.

    Scott

  • #2


    Hmm, not exactly sure here, but there is a bug in the current PmmC that 'Just might' be causing what you are seeing.

    Hopefully the new PmmC will be available in the next week or so.
    Mark

    Comment


    • #3


      Let me be a little more specific about my question.

      In most every example of code in the 4D Systems documentation, when the uSD card is used, there are a couple of lines where the uSD card is initialized, and the code loops after printing an error message to the screen.

      I would have thought the init() call would attempt to re-mount the uSD card. I'm seeing no evidence of that...it loops in that error, forever (even twiddling the reset pin seems to just start the uSD error again). If I power-cycle the video module, however (with no change in the health of the battery...just unplug/replug), there's a pretty much 100% chance that the video module comes up fine.

      So, I get into this uSD card init error loop when there's *maybe* a marginal battery, but there's seemingly *nothing* I can do to recover from the situation (short of a physical power cycle...even toggling the reset pin seems to have little effect). I'd really like to do something like setting a timer for 120 seconds in the future and try booting again when it expires, or leave the display off for several seconds and do a deep reset/retry to init the uSD card.

      Does this seem like something the pmmc bug is about?

      Thanks,

      Scott

      Comment


      • #4


        Yep, that's exactly what the PmmC bug is about
        Mark

        Comment


        • #5


          That would be fantastic, if there was a way to recover from this zombie-boot situation. Right now, it's holding up the completion and shipping of a project that includes dozens and soon hundreds of your modules.

          We'd been trying to work a hardware solution to the issue...looking for ways to predict/detect when the video module was going to have sufficient current to boot properly, but that has turned out to be difficult. According to my hardware engineer, for some reason, the video modules appear to be exceedingly sensitive to the impedance of the battery/charging system. This means that the situation is markedly different when there's just a battery present vs. when there's a battery and charger (with the latter mysteriously being more problematic, even though the charger alone can deliver more than sufficient current for booting).

          I suspect the problem is rooted in transients...that the time constants of the various components are just not fully compatible when we go from zero to booting current very quickly. If I could get a real re-initialize and re-mount of the uSD after/while the module was running, I think this situation would be one that our system could seamlessly recover from.

          Fingers crossed. I know you can't say for certain, but it this pmmc release looking like it'll really be out in the week-ish timeframe? Looking back over older forum posts, I've read about long delays in some earlier ones. I just need to have moderate certainty to tell the stakeholder in this project that we're close (and that waiting for this fix is wise, instead of beating our heads against the wall trying to figure out how to do it in hardware).

          Thanks so much!

          Comment


          • #6


            I know you can't say for certain, but it this pmmc release looking like it'll really be out in the week-ish timeframe? Looking back over older forum posts, I've read about long delays in some earlier ones.
            Err, yeah , hopefully those days are behind us. We are still aiming to release this week, it's in 'final' testing.
            Mark

            Comment


            • #7


              Ok, so I tried the new pmmc, with no change in the behavior I'm seeing.

              Here's a code fragment:

              while(media_Init() == 0) // initialise and test the uSD card
              gfx_Set(CONTRAST, 4); // so we can read the error msg
              gfx_MoveTo(0,30);
              print("uSD didn't init properly.");
              gfx_MoveTo(0,45);
              print("Waiting a minute");
              gfx_MoveTo(0,60);
              print("before trying again.");
              pause(3000); // time enough to read the error msg
              gfx_Cls();
              gfx_Set(CONTRAST, 0); // minimize consumption for the wait
              pause(60000);
              wend

              I was hoping that media_Init() would eventually eventually return a successful value and my program would operate properly.

              To review the situation, I have a system where there might be insufficient current capacity at the very outset to initialize the uSD card. Perusing the forums has indicated that this is a very common problem. What I'm hoping for is some way to, from a running instance of the module, kick the system hard enough to get it to mount the card reader.

              This condition can be simulated by connecting the module to a current-limited bench supply at 5V, ramping up the current to where the module boots and then cranking up the current until it could also run the uSD. It seems like many of us need to be able to deal with this zombie-module condition in software.

              Any suggestions or ideas? Hitting the reset line is also unsuccessful. This loop continues to report failure *well* into the phase where the battery has *plenty* of capacity to run the module and the uSD card (it'll run for 8 hours continuous after the disconnect/reconnect). The *only* way I've found to recover is to disconnect the battery and reconnect it. This is not an acceptable thing for the users of my system to have to do.

              I'm also not terribly pleased with the possibility of needing to include a deep reset switch that'll interrupt the battery voltage, because that means I need to run quite a few mA through a tiny normally-closed switch on my power-condition circuitry, which seems like a recipe for eventual reliability issues.

              Comment


              • #8


                What make/size of uSD card are you using?

                Can you try another?

                I'm wondering if the uSD card is internally shutting down when it sees an 'unacceptable' voltage for a period of time.

                It might be a bit hard to do, but what if you remove and reinsert the uSD card after it gets into a locked state? (I say this might be hard to do because some cards have a fairly large internal capacitance and inserting them with marginal power can cause the module to reset)
                Mark

                Comment


                • #9


                  The uSD card is a 2GB Samsung. I have a few 2GB Kingstons here too. I swapped the uSD card out and still see the same behavior.

                  One thing I wanted to ask...your note hinted at what's maybe the root cause, here. The problematic reboot is *not* an intentional one. When I plug the module into the charger (no matter how dead the battery is), it immediately starts and will run flawlessly as long as I keep the device "awake" and the display at normal brightness. If the device drops to zero brightness (to speed our charging time in this quiescent state, triggered by the ), then there's a good chance, as the device be waking back up to full brightness (which I ramp up slowly, for aesthetic reasons), that it will decide to reboot. These are the reboots that cause problems.

                  Throughout this "shallow sleep" period (video playback to a dark screen), the uSD card is being accessed in the background. In normal wake-up from this state, I see the video playing wherever the frame pointer happens to be at that moment. In spontaneous reboot, I see the brief flash of the last frame the video module loaded (not the same frame each time, by the way), and then either the video plays (yay!) or I get the error message from the code fragment above (boo!).

                  If I sneak up, when I've gotten the error message, and eject/reinsert the memory card (during the 60-second pause before the next re-try), the next time through it seems like the module will start properly. I'm not ready to say this is always the case, because I've only tried it a few times.

                  Throughout this period, the voltage out of the boost converter appears to stay at 5V. We don't seem to be at a voltage level where we're triggering cut-outs of any portion of the power-delivery system.

                  What are these events that you've seen cause spontaneous module reboots? Are there things than can be done (additional capacitors on Vin, perhaps) to eliminate the possibility of this occurring?

                  Everything works perfectly off the charger. The battery voltage starts at 4.13 or so, and the module works great all the way down to 3.5something volts. The charger can supply an amp. The LiPo charger module can deliver 280mA. The boost converter can churn out 300mA, if needed.

                  Thanks for any additional suggestions or ideas.

                  Comment


                  • #10


                    Hmm, what was fixed in R24 was the ability to 'cold' initialize a uSD without Goldelox being reset. (It's rather hard to describe) I thought that that was was what you were seeing and that all would be good now, oh well...

                    The uSD card is a 2GB Samsung. I have a few 2GB Kingstons here too. I swapped the uSD card out and still see the same behavior.

                    Hmm, unfortunately there are probably exactly the same, early Kingstons that I have examined where made by Toshiba, the newer ones are made by Samsung.

                    Could you try a SanDisk?

                    Throughout this "shallow sleep" period (video playback to a dark screen), the uSD card is being accessed in the background. In normal wake-up from this state, I see the video playing wherever the frame pointer happens to be at that moment. In spontaneous reboot, I see the brief flash of the last frame the video module loaded (not the same frame each time, by the way), and then either the video plays (yay!) or I get the error message from the code fragment above (boo!).
                    Could you perhaps disable this video playback to a dark screen, to see if it changes anything? (i.e. no uSD access in "Shallow sleep")

                    Does this happen with some uOLED-160s only and not at all with others?
                    Mark

                    Comment


                    • #11


                      I mis-spoke (or mis-typed). The first uSD card was a San Disk, the second was a Kingston. They're not identical, at least as evidenced by the fact that, after doing the exact same format operations, the videos end up at lightly different offsets. I currently don't think it's the fault of the card(s), but should I be shopping for something specific? Are 1GB cards the best bet?

                      It also happens on multiple video modules (i.e., all of them). I have about 50 of the boards around at this point...just waiting for this last bug to get swatted.

                      I've changed the code so that I'm not accessing the uSD card if there's nothing visible on the screen, and I'm now waiting for the battery to go dead enough for the next round of testing. I could rig something up that drains them faster, but I need for this testing to be as completely representative of real-world use as I can.

                      Cheers,

                      Scott

                      Comment


                      • #12


                        Ok, so disabling the video playback (i.e., not reading from the uSD card) during periods where the screen is dark doesn't solve the problem...there's still some number (more than half) of times when, rather than the board rampings back up from dark, it reboots. It seldom is successful in those early reboot attempts (i.e., it doesn't mount the uSD card ). This time I am letting it sit on the charger for a long spell to see if it will stop doing these spontaneous reboots if the battery gets healthier before the board tries to wake up from dark.

                        I'm also going to power it with a bench supply once or twice, just to make sure it's not some odd interaction with the wall wart.

                        I've now had a few more chances to see whether removal and re-insertion of the uSD card gets it out of the init loop. I've found that it does. I was under the impression that this condition was *exactly* what R2.4 was going to address...a hard reset of the uSD device/card.

                        So, I'm seemingly still in a situation where either 1) the uSD card has to be removed and re-inserted or 2) the module itself has to be power cycled.

                        Comment


                        • #13


                          I'm wondering if the reboots are from brownouts, I was more concernded that you couldn't get the uSD reinitted after them.

                          But you seem to have confirmed that if you remove and reinsert the uSD you can, so it more or less means that the uSD card is locking up after a 'similar' brownout.

                          Can you plot the voltage 'drop' to see at what point everything starts to go belly up?

                          I'm happy if you tried SanDisk.

                          There is a bug in the Samsung cards and I was just wondering if that was somehow causing what you are seeing, but if it also happens with SanDisk then that eliminates that possibility.
                          Mark

                          Comment


                          • #14


                            I'll put the thing on a scope and look for any voltage drop/droop, but the module is downstream of a closely-regulated boost converter. Maybe we can see something, but we've checked before and not observed much wavering.

                            Before we got better instrumentation, we'd used a very small resistor in series with the module so we could watch the current (by watching the voltage drop across this very low value (1 Ohm, if I recall correctly) resistor). What we saw was very surprising...the module would not boot, even when connected to a very stiff bench supply, ready and willing to supply high current. My electrical engineer described the video module as being highly (hyper) sensitive to the source impedance.

                            We'll look further and see if we can provide more information about the symptoms at the brink of the issue.

                            Much as I'd like to address the cause, I'm equally curious about what can be done to treat the resulting symptom. I'm not seeing much information about SD card behavior that would result in this sort of lock-up, and what's accomplished by removal and re-insertion of the uSD card (i.e., can we simply interrupt the power or ground momentarily to accomplish the same ends). Can you shed any light on this matter?

                            Any other ideas?

                            Comment


                            • #15


                              Um, Removing reinserting is 'just' to temporarily remove power to the uSD card.

                              Of course other things might happen because of that.

                              eg different uSD cards seem to have different values of C across the power rails. Some appear to have nothing, some 'appear' to have 10uf (sounds impossible doesn't it, that's why I put appear in inverted commas).
                              Mark

                              Comment

                              Working...
                              X