I have an android phone.  I have lot’s of local music (~17,500 songs) on a music server.  I’d like that music to be more portable and to keep my collection more up to date.  I use Pandora often as an automatic DJ but Pandora often plays what it has available which doesn’t always overlap with what I want to hear.  I’ve tried Amazon MP3 and though it is convenient, there are too many limitations with a large music collection.

Enter Google Music.  They will handle 20,000 songs for free.  You can easily purchase songs and download to your devices and in my case my music server.  Content I buy elsewhere can also be placed on my music server and synced with my Google Music using their Music Manager software.  This sounds ideal for my situation.

So far I’ve uploaded my collection and made some attempts at using the service.  Uploading took several days on a 10mbps upstream connection.  The first issue is the playlist system, what is playing and how to control this is not particularly intuitive.  I hope this will improve.  More concerning is some new Music I purchased.  It shows up in my music server after a download.  It shows up on the Google Music website.  However, when I use the Google Play Music App, I am seeing some failures.  One song in particular is missing while all others are fine.  Google had me go through a series of corrective actions which I performed.  Their steps culminated in flushing the data and cache which did not resolve the issue.  Will other music go missing?

I’ve also noticed search results on my phone differ from those on the website.  I’ve clicked the “keep” button in the Google Play Music App to store some songs on my phone locally.  Apparently wiping your data also removes these songs unlike Amazon MP3 which does a hard download to a general music directory.

Overall, Google Music meets my needs by specification but the execution and some small issues leave me without full confidence and nagging use complications.

My company has been actively using an Allworx 6x since 2008, during which time we have always been using a SIP trunk for calling and have been with bandwidth.com as our primary provider.  There are so many problems we have encountered over the years with so many caveats that it won’t help to review the old information except to say we have burned too much time working with Allworx idiosyncrosies over the years.

Pros:

1. A solid, local hardware solution with programmability to work with hard lines, SIP trunks and a great deal of programmability on the phones.
2. The phones have programmable buttons with multiple colors such that you can see messages, other lines busy or a variety of other functions.
3. Many improvements over the years have led to a mature architecture that does 95% of what we want to do.
4. The handset call quality is generally good and the basic phone function allows for an efficient workflow with call staff and regular staff.

Cons:

1. Redundancy is an afterthought and support is little help.  I want to buy a solution using a SIP trunk that can failover from one connection to another easily with a high quality, multi-wan router sitting in front of the Allworx. Due to the way the Allworx is programmed and it’s SIP packets are handled, a solid solution does not exist.  The result is unreliable phone service if you have an unreliable connection.  If this one issue were dealt with, my happiness with this system would improve greatly.
2. The phones are outlandishly expensive for the quality you get.  Many phone models are not duplex despite marketing claims.  I spent more money adding phones for several offices this year than some entire systems cost with more phones in total.
3. Poor documentation for various features.
4. When we bought this system in 2008, we were using ringcentral with many issues.  As cloud services have matured, the need for a local PBX has diminished.  The benefits of a local PBX do still exist but they come at greater cost that must be weighed more carefully now.

Conclusion:

The Allworx system is generally competent and worth a solid look.  Were I starting my company today and faced with the same decision I was faced with in 2008, I believe I would choose a cloud based service.  The lack of redundancy, difficult to navigate or unhelpful support and high cost (once phones and features are included) prevent the Allworx 6x from competing with now mature cloud based phone systems.  During this same time, the cloud services have themselves matured and now offer similar or better reliability and features at a lower price point.

With traditional hard lines instead of a SIP trunk, the Allworx 6x would gain reliability but lose flexibility at which point my conclusion remains the same.

Yesterday, one of my sites went down while using a Cradlepoint MBR1400.  This was especially odd given that the area did not have an internet outage and because this site has a Comcast primary and 4G secondary connection.

I was not on site and the local staff chose to reboot the router which resolved the issue short term.  The question remained, why did the connection fail with primary and secondary connections that were up?

The answer came from the the failure check settings in the MBR1400.  Both the primary and secondary connections were set to check 8.8.8.8 for failure testing and failback was based on time.  8.8.8.8 (google dns) went down causing the primary connection test to fail so the cradlepoint switched to the secondary connection.  The secondary connection test to 8.8.8.8 also failed causing the MBR1400 to re-try primary.  Essentially at this point the system entered a logical loop in which it stayed stuck until a reboot.  The total downtime was approximately 25 minutes (unacceptably poor).

There are two parts to the resolution here.  The settings need to be altered as they were programmed poorly (though the documentation on this is lacking).  After some testing and communication with Cradlepoint I will update which settings we found to be effective.  The second part to the resolution requires a fundamental change from Cradlepoint.  Simply put, you cannot test a single ip otherwise you may end up with a false outage as we did.  Even if that ip has Google reliability.  The failure check testing should be using 2 to 4 ip addresses with programmability for how quickly a positive result is active upon.

So, please improve your failure check Cradlepoint to easily prevent false outages for the market you serve.

Update: Per recommendations from Cradlepoint, I changed the settings per the images below.  The results were much improved.  45 seconds and several lost packets to failover on a primary line disconnect. ~5 seconds and one lost packet on primary modem power failure. Failback in both cases lost only one packet and didn’t come up until the cable modem was ready.  We also worked with comcast to improve the modem cycle time.

While these improvements are a huge, there are still issues.  The Cradlepoint problem of testing only one destination for our primary wired connection is a clearly identifiable point of failure that largely negates the redundancy benefits.  The failover event could be improved on our primary line disconnect test case, bringing the time down from 45 seconds and many lost packets which we know competing hardware can achieve.

Primary, wired connection.

Secondary, 4G connection.

We purchased these upgrade packs with a poor description on the Newegg website.  Basically they only upgrade from Win8 to Win8 pro and are not an upgrade from a prior OS to Win8 pro.  When we caught the error, we wanted to send the parts back.  The MS policy on this is simply to go $&^! yourself and Newegg backs them up on this.  There are a lot of people similarly effected due to the way the product is described.  We did a chargeback to Newegg who automatically canceled our account.  The net result is that American Express gave us a refund at their cost as we pump tens of thousands of dollars through the card per month.

In the end, we were left with a sour taste.  We restored our account with Newegg but after a few similar parts failures that ended up costing us time and money, we avoid Newegg.  Now we use Newegg for research, then buy on Amazon or some vendor who stands by their products so we don’t burn time.  We don’t return parts often (far below 1%) but when we do, we don’t want to burn time and effort on the process needlessly.

If you look at reviews for these same items on various websites, the ratings are extremely poor.

http://www.amazon.com/Microsoft-Windows-Pro-Pack-Upgrade/dp/B0093H8W4W

Microsoft Windows 8 Pro Pack (Win 8 to Win 8 Pro Upgrade)

Upgrade your Windows 8 device to Windows 8 Pro with Pro Pack

http://www.newegg.com/Product/Product.aspx?Item=N82E16832416563

I recently had a ReadyNAS Pro Business Edition fail and lost about 8TB of information.  Approximately 3TB of this data was held with Crashplan and was critical.  This would take about 25-50 days to restore, necessitating a “Restore to your Door” service with Crashplan.  In short, you can call them and they will send you a drive at some cost that depends on your plan. A co-worker suggested I sneakernet the data which led to a fun comic about sending massive data over the internet.

Luckily Crashplan has an adopt a computer option which is intended primarily for replacement devices.  This means I won’t have to go through all of the settings and options again after the restore process is complete, this ReadyNAS should pick up where the old one left off.  The quick steps in short are to install crashplan, restore the files then adopt the old computer.

It took me some time and effort to install Crashplan on a ReadyNAS the first time around a few years ago and I intended to publicly document the process but never did.  This time around I was happy to see that a forum member at the ReadyNAS forums created a fantastic step by step document here.

Once crashplan was installed, I used the crashplan gui at my local station and started the restore process. My local station also has crashplan installed, so I keep the “C:\Program Files\CrashPlan\conf\ui.properties” file with a shortcut on my desktop to change the port and added the following comment to the file “for local access, servicePort=4200 for remote access.” Once the restore process is completed, I will adopt the old ReadyNAS profile using these instructions.

I’ve put up a number of posts recently regarding our ReadyNAS which largely stem from the fact that I am working on it extensively due to a total failure.  I’m just going to list the timeline for the failure scenario and what happened.  At this moment I am very glad I chose the Business Edition rather than the Pioneer Edition due simply to the 5 year Business class warranty which was a necessity and convenience recently.  The warranty certainly paid for itself.

1. ReadyNAS Pro Business Editiona was purchased in August 2010 with 8 WDEADS drives with TLER on and parking disabled (6 installed with 2 hot swaps). ReadyNAS was responsive with lots of capacity and high speed.  Great!
2. Played with more and more utilities and systems.  ReadyNAS was slowed down over time and some services were removed at various points.  Firmware updates were performed regularly. Drive use was somewhat extensive due to continuous video archiving from security system.
3. Dropbox and Crashplan were installed, slowing system down and eating memory.  Sometimes the ReadyNAS https admin panel was unresponsive or unreachable.
4. ReadyNAS lack of responsiveness became dramatically worse.  A FAN CAS message failure dialog showed up on health screen and showed a yellow warning with a correlating temperature discrepancy on AUX. At this stage, any changes made in the admin panel were not saved and variables and modules which were previously set became unset or failed directly.Error Message Below:“Disk 1 WDC WD20EADS-32S2B0 1863 GB , C / 32 F , Write-cache ON OK
   Disk 2 WDC WD20EADS-32S2B0 1863 GB , C / 32 F , Write-cache ON OK
   Disk 3 WDC WD20EADS-32S2B0 1863 GB , C / 32 F , Write-cache ON OK
   Disk 4 WDC WD20EADS-32S2B0 1863 GB , C / 32 F , Write-cache ON OK
   Disk 5 WDC WD20EADS-32S2B0 1863 GB , C / 32 F , Write-cache ON OK
   Disk 6 WDC WD20EADS-32S2B0 1863 GB , C / 32 F , Write-cache ON OK
   Fan SYS 958 RPM OK
   Fan CPU 2136 RPM OK
   Fan CAS 0 RPM Out of Spec
   Fan RPM OK
   Fan RPM OK
   Temp SYS 49 C / 120 F [Normal 0-65 C / 32-149 F] OK
   Temp CPU 13 C / 55 F [Normal 0-60 C / 32-140 F] OK
   Temp AUX 3 C / 37 F [Normal 0-0 C / 32-32 F] Out of Spec”
5. The lack of free memory and perhaps memory issues were thought to be at fault.  Offline memory testing was performed and the memory tested fine.  New memory was eventually purchased (PSD24G8002, see other post for details) and installed.
6. The ReadyNAS continued to fail, be unresponsive and eventually became dead to all outside inputs except for a periodic ping responses.  OS Re-install was performed at the direction of Netgear support.  No actions remedied the problems.  Offline disk testing was NOT performed but should have been in retrospect.  Factory Default was not desired because we did not want to lose the information.
7. A new ReadyNAS chassis was sent out.  The drive holders have a slightly different design.  Offline memory testing was performed and all was well (with the oem memory).  Drives were moved to the new chassis.  This unit was also unresponsive.  A factory default was performed after it was determined the critical data could be salvaged from other sources (Crashplan & Dropbox)
8. After factory default, disks 1 and 6 were found to be faulty as the ReadyNAS performed a disk test on RAID array creation.  Later extensive testing at a computer found disk 1 to have excessive bad sectors and no problems with disk 6.  Both WD20EADS drives were warranty replaced by Western Digital.
9. An XRAID2 was rebuilt with the 4 remaining drives.  Several days of memory testing with the new Patriot RAM were performed on both the old and new ReadyNAS devices with 100% pass rates.  Patriot RAM was installed in new ReadyNAS.
10. Warranty drives came in and were installed.  ReadyNAS completed building the array.
11. Patriot Ram was installed in new ReadyNAS and all is well at the moment.  A review reveals that weekly file consistency checks were being performed as well as regular RAID scrubbing.  Neither check caught the errors.  A new procedure has been put in place where the ReadyNAS is brought offline and offline disk testing is performed monthly.  In addition, every 6 months all drives are removed and extensive low-level testing performed at another workstation to validate disk function.

In the end, the errors seem to have stemmed from gradual malfunction of the ReadyNAS which masked drive issues.  It is possible early drive issues began to cause the ReadyNAS problems.  In either case, the failure is disconcerting.  8TB of data being archived was lost.  Fortunately, we had the foresight to have this data concurrently duplicated at Dropbox and Crashplan which turn out to be the saviors here.

We purchased an Allworx 6x in 2008 and have varied results with the system over the years. Very recently we had some intermittent rebooting issues we believed to be caused by a 5 year old Compact Flash card which has accumulated many write cycles (256mb is what came with the system at the time).

We asked our rep what cards would work and he felt it should be a type ii CF card.  At first the reference seemed to be to the CF type itself but it turns out it is a reference to the speed types offered by Sandisk.  Sandisk offers different speeds such as Ultra (ii), Extreme (iii) and Extreme Pro (iv).  Apparently there are potential write issues with the iii or iv so we went with the maximum we saw currently supported by Allworx in that family which is a Sandisk Ultra II 2GB (a massive upgrade over our 256mb older, slower card).

The simplest information I found was actually in their marketing information to validate this.  On Page 6, there is the reference “Compact Flash Slot – Accepts Ultra 2 SanDisk Compact Flash card” in the pdf below:

Allworx 6x_Brochure

In the end, it cost ~$13 and worked.  I bought some spares because of the extreme low cost.  We do backups to a server (whose automation is broken for reasons we haven’t investigated) but we are able to do manual backups.  To get this process to work, I had to install Office Safe Backup 5.3.1 (click here to download) on my laptop and copy our backup directory to it to restore from the backup.  I had to set my laptop on their network with a direct connection.

Installing a new Compact Flash Card is described in the installation guide below on Page 8:

Allworx 6x Installation Guide

The backup and restore process is described in the Administration Guide below on Page 75:

Allworx_Administration_Guide_723x

In the end, the process worked.  Getting the new compact flash card installed and operating was simple but it was the restore process which took a little more time and was less sensible.  It was needlessly convoluted spanning multiple documents, steps of validation and having to do everything from a laptop when the restore process could easily have been accomplished like a router flash upgrade.

We use our ReadyNAS Pro Business Edition for work purposes which is archiving video, some of our servers and some critical data.  Part of our redundancy plan is to use Dropbox for files which need to be used across various employee devices and crashplan for data that does not need to be distributed but still needs to be reliably archived.

After installing these services, our ReadyNAS became quite slow and was not as responsive as desired.  We saw  that our memory usage was quite high and felt this was a critical cause.  As it turns out, it was a combination of this and a hardware failure on the ReadyNAS.  Some services are RAM hungry as well for home users like TiVo & music serving options.

The ReadyNAS hardware compatibility list does NOT list RAM so this is left up to forum users.  There are a large number of posts with varying success.  We purchased and extensively tested Patriot Memory Signature DDR2 4GB CL6 800MHz DIMM (PC2 6400) PSD24G8002.  This RAM proved successful with very simple installation (just remove the left side cover with two screws, remove the old ram and insert the new ram then replace the cover).  Be certain to perform extensive memory testing! (Click here to see how to get to the boot menu, then select memory test and let it run overnight).

In August, 2010 I purchased a ReadyNAS pro business edition and wanted to get the most storage for the dollar.  WD20EADS drives from Western Digital looked like a great option because despite being consumer drives, two critical RAID requirements could simple be enabled on these drives.

The first is feature is to disable a head parking feature with a utility called wdidle3.  If left alone, this feature which normally saves power on a desktop, will lead to an excessively high Load Cycle Count in a RAID environment causing potential failure in the long term.  Good information about this abounds on the internet so I won’t cover it specifically, here is one opinion (link) .  With aggressive head parking disabled, my LLC count is below 300 for all of my almost 3 year old drives which are on 24×7 which is excellent.

The second is to enable TLER.  At the time of my build I believed TLER to be critical and currently no longer believe it to be critical in this RAID environment.  Here is a short breakdown. In brief, it provides for error recovery for a hard drive that does not respond right away in a RAID array to prevent the array from dropping the drive.

Moving on to the functional, I turned on TLER (7 second read and write function) and disabled the aggressive head parking.  I did it with the tools below.

fdoem-iso (click to download iso, 32bit only)

This is an iso with wdidle3 preloaded and TLER capability preloaded.  Just write this iso to a CD or use tools (not described here) to boot a USB stick with this iso.  Once you boot, type “wdidle3 /d” or in some cases (if this first command does not work) “wdidle3 /s0” to disable aggressive head parking.  Some drives will not allow these options and you can alternatively run “wdidle3 /s300” which will only park the head once every 300 seconds which will at least greatly alleviate the issue.  “wdidle3 /r” will report the status.  To turn on TLER, just type tler-on which enables TLER with a default 7 second read and write.  You may run tler-scan to see the status of TLER on your drives.  wdspinup is also on this iso which I did not use but gives the user spinup time control.  WDTLER.exe and wdidle3.exe as well as hdaccess.exe are also included.

fdoem-cdmaker (click to download tool, 32bit only)

This is the tool I used to create the iso.  You can add other functions or tools to the CDROOT directory to customize your own iso.  Just add what you want to the CDROOT directory then run makeiso.bat

fdoem-64bit (click to download tool, I have not tested 64bit)

I also found this tool here. I understand this is the same tool that will now operate on 64bit systems.

Now, onto the main reason for my post.  I recently had two WD20EADS drive failures concurrently which stemmed from one bad drive and a failing ReadyNAS.  I will cover the failure and resolution in another post but I lost all my data on my ReadyNAS (~8TB worth).  Fortunately, I have a backup plan which teams Dropbox and Crashplan on the ReadyNAS as well as client machines which led to very limited actual data loss.

For anyone with these drives, my recommended procedure going forward is to take the ReadyNAS offline at least once a month and perform an offline disk test (click here to see how) .  The reset button is on the back of the machine in a small hole above the rj45 connections.  I recommend every 6 months, shutting down the ReadyNAS and taking the drives out for a full, extensive scan with a tool like WD Diagnostics or similar free tools to validate full function as the ReadyNAS offline tests are not low-level scans.

The WD20EADS drives were still under warranty though not by much.  I was able to get replacements which are WD20EARX drives.  I did have to run the wdidle3 tool on the near EARX drives but TLER cannot be enabled.  Apparently WD caught on to what users where doing and functionally disabled the possiblity of TLER being enabled on the green drives.  So far the EARX drives are working and run about 5-8 degrees Fahrenheit cooler than the EADS drives.  I hope the lack of TLER will not prove problematic.

In the end, the cheap WD drives cost me more money and time and effort than they saved.  On initial purchase, one of them was DOA.  I believe no cost reasonable enterprise drives were available at the at 2TB time but going forward I will be using only drives on the ReadyNAS Hardware compatibility list .  At the moment I will use the existing drives but with the new testing procedure in place described above.  Going forward, I only plan to use enterprise drives such as the RE4 WD series or Constellation Seagate series.  Were I to choose today, I would get 3TB Seagate Constellation ES2 drives ST33000650NS .

UPDATE (2/17/2013):
I ended up purchasing 6 ST33000650NS after the replacement WD20EARX drives showed ongoing ATA failures and the old WD20EADS drives had additional failures.

Oh how I hate Comcast, let me count the ways.  Beyond frequent drops, poor service, terrible billing and mass confusion, hysteria and poor technical understanding at Comcast’s technical support I have found a new reason to dislike this corporation with a monopoly in my region.  Specifically, I installed Comcast at a commercial facility recently and used my laptop as the tech did not bring equipment to perform the installation (at the wrong date and time).  So, he requested I install the Comcast “software” to perform the installation.  I did this and we were able to get everything working after some time with him on the phone with Business Class support to get our fixed IP configured.

The next time I used the laptop, I found that all browsers had been unknowingly and unwillingly commandeered by an Xfinity default home page and Xfinity default search.  Seriously Comcast?  On to the removal.

Firefox Removal
For the homepage change, go to the Firefox button in the upper left or Tools if you are still using a menu bar.  Choose options from tools or options then options from the Firefox button.  In the General Tab, you can choose what firefox does when it opens.  Xfinity will have set it to their link and you simply change this to nothing or back to default.

For the search change, click the down arrow in the upper right search box where you see the Xfinity logo.  You should see Xfinity listed as well as an option to manage search engines.  Choose manage search engines, select Xfinity and remove.

Google Chrome Removal
For the homepage change click on the three bars at the top right of Google Chrome which is their menu.  Then choose settings from this menu.  You will see an option that says on startup.  You can click the set pages blue text to change the default web page or open new tab or continue where i left off.

For the search engine change, click on the manage search engines button under search.  Click on the search engine you want as default then click make default.  Select Xfinity and click the X on the right to delete this service/option.

Internet Explorer 8 Removal
For the homepage change, click on the gear icon in the upper right to bring up the settings, then choose internet options.  On the general table there is a homepage box where you can use default, blank or change the Xfinity option.

For the search engine change, on the general tab there is a change search defaults row with a settings button.  Click this and then remove the Xfinity system and choose your desired default option.

These methods are changing with time and I didn’t choose to do screenshots in this walk-through but I hope it will help some people who might not otherwise know how to remedy this Comcast blunder.