Intermittent Shaw Internet Outages: What is going on?

by Chris Roehrig
croehrig@house.org

As many folks on Mayne Island know, Shaw internet has been frustratingly intermittent for the last week. Last Tuesday (July 31), Shaw's undersea fibre optic link between Pender and Mayne Island was severed which cut off cable TV and internet service to Mayne and also to Galiano (which is fed via another undersea fibre cable from Mayne).

Service was partially restored the following day using a temporary link, but it has been flaky, with intermittent 2-5 minute outages, sometimes several times per hour which makes it frustrating if not almost impossible to do some internet activities.

I had heard that this temporary link was a microwave radio link, but that turns out not to be the case.

I spoke to Greg Pultz, the VP of Operations of Shaw to find out what is taking so long to get reliable service restored. He is very aware of the situation (and was actually in Victoria when the cut happened) and was quite forthcoming with information.

Greg told me that they sent divers down right away, but the cable was completely severed and they could only find one half of the cable. If they had found the other end, they could have spliced it and normal service would have resumed quickly. As it is, they will lay another fibre optic cable which requires governmental approval from the Dept. of Fisheries and other agencies. That is mainly what is responsible for the delay in restoring the main cable.

The current temporary link is actually another old undersea fibre optic cable that was damaged in an earlier incident.  The cable wasn't completely severed and they found that there were still two optical strands that were intact, and that's what is currently providing connectivity. However, the strands are degraded and that is what's contributing to the intermittent outages.

There will shortly (in the next few days, probably early next week) be a microwave link set up between Pender and Mayne. They already have the necessary permits, cooperative landowners on both sides, and are now just waiting for the arrival of hardware. This link should resolve the intermittent problems we are currently having, and will be a permanent backup link for Mayne once the new fibre cable is laid (and provide increased capacity if necessary).

In the meantime, they are still tracking down the cause of the intermittent outages which they believe is caused by some noise getting into the system.

The distribution network on Mayne and Galiano (the "plant" as they call it) is a hybrid fibre-coaxial network. The cable coming to your modem is a copper coaxial cable, but the main trunk cables are optical fibre. The transitions between optical and copper happen at "nodes" (there are 3 or 4 different nodes on each island) and happens entirely in the analog domain (which was news to me), which means that any signal loss from the degraded optical strand is carried through right to the modem. When the signal drops below a certain threshold, the modem goes offline. Modems that go offline and come back online due to marginal signal strength (as we are currently seeing) is called "flapping" which is a term you may hear when you phone in to Shaw technical support. (Modems can also go offline if the signal strength is too strong for the receiver; attenuators can be used at your modem to reduce the signal strength to a suitable level).

There are a lot of potential causes for noise and loss of signal strength and they are cumulative. For example, the Shaw tech that came to my place a few days ago replaced corroded cable ends on the poles – there were no rubber grommets to protect them from the weather and moisture seeps in and slowly corrodes the core of the copper coax cable resulting in a loss of signal strength. When the previous owners rolled out cable internet on Mayne, they did it on a shoestring budget, and there are a lot of legacy situations that Shaw is improving as they go along. (I'm so thankful that Shaw bought Pender Cable and now has the resources to make these improvements. I think they are doing an amazing job considering how marginally profitable it can be to do business on the Gulf Islands.)

While replacing my cable ends improved my signal strength by a few dB, it didn't have any noticeable effect on the intermittent outages. I've been comparing notes with some other people on the opposite end of the island from me and noticing that their modem "flapping" is occurring at exactly the same time mine is, so I'd guess that the main source of noise is those damaged strands and that there may not be much that can be done about it until the link is replaced.

I expect that when the microwave link is up and running next week, the intermittent outages will be resolved, though we many still have reduced capacity (i.e. download/upload speeds) until the new undersea fibre is in service.

One thing Greg did say is that they were not aware that the intermittent problem was so widespread since they had not gotten many calls about it. If you are having problems with the internet, call Shaw tech support so they can track how widespread the issue is.

I'll post more information as I get it.


UPDATE Tuesday, Aug 14: I spoke to the Shaw Regional Manager for Vancouver Island today. He said they isolated Mayne and Galiano onto separate fibres over the weekend and had crews there over the last couple days tracking down the noise issues (which seem to have gotten much better).
 
Since they are getting the noise issues resolved, they have backed off of the urgency of the microwave link (which will still be set up as a backup link).

UPDATE Sunday, Aug 19: Here's an update I got from one of the senior techs working on the intermittent noise issue. He makes an interesting point that if you have an unterminated cable (say a disconnected TV), it can act as an antenna and can feed back into the entire system.

"[T]he intermittent flapping is caused by some noise seemingly originating in the Bennett Bay Area. Due to the very intermittent nature of the noise, it’s incredibly challenging to be at the right place at the right time to actually spot the noise and then attempt to further isolate it to various roads, houses etc...It would be far easier if it just stopped working , that way we could then easily track it back to its source. The reason this is now impacting all Mayne Island customers is because we had to combine Mayne Islands 3 fibre nodes onto 1 fibre rather than its usual 3 fibres. When each fiber node has its own fiber, if there is noise in Bennett Bay for example, it’ll only affect Bennett Bay, not Mariners Way, and vice versa. By combining 3 nodes, temporarily, we’ve increased the noise floor, so a small noise issue that may never even had shown up in all of our regular maintenance, now becomes far more critical. We have been over every day this week, hardening the plant and hoping by some fluke that we get to see the noise while we are standing in front of a piece of equipment. To make matters even more complicated, coincidence can also play a part in tracking down such a sporadic issue. Without going into too much detail, it’s virtually a needle in the haystack type of scenario. We will find the issue, we just ask for a bit of patience.

"At this point the 'noise' could be coming from anywhere, a customers drop, a broken mainline cable, we're just not sure. We think we may have narrowed it down to the Bennett Bay area, but due to the nature of the noise, even this assumption may have been made from sheer coincidence.

"Customers could make sure that all cable ends are screwed on tight to their DCT's VCR's, TV's and Cable Modems. If there are broken ends, cables that have been cut or chewed, they should call us and we'll do a service call to make sure this isn't impacting the entire Mayne Internet service. 1 cable in a customer's home that is open and grounding out on something that can act like an antenna could in theory sink the whole ship."


UPDATE Monday August 27: Service has improved quite a bit over the last few days, but I'm still seeing occasional 5 minute outages a few times a day.

It turns out the issue wasn't noise entering the plant after all:

"We did make some changes late last week that have lessened frequency of the internet issues but we knew it would just be another step in the restoration of services. The interference that is affecting your service is called optical beat interference and is being caused by us having to combine the returns from 4 nodes - Galiano (1 node) and Mayne (3) - together on the fibre path back to the hubsite on Pender. What we did last week was to separate the nodes into pairs to lessen the amount of transmit power on the fibre. We will be moving to what will be our final configuration of our plant on Mayne and Galiano that will allow us to separate all the returns by using tuned transmitters at all the nodes that will put each node at a different optical frequency and that will eliminate the OBI issues we have been seeing.

"We are building a microwave system but it will only be a backup to the fibre we have in place now. The microwave has less bandwidth available on it than the fibre but makes a great failover path in the event of a failure of the fibre we are using now. A new submarine cable will be placed but with all the regulatory requirements we need to meet before proceeding it will be some time before that project is complete. We have been putting a lot of effort into trying to improve the service (and I know it does not always seem like it) but I hope by this time next week I can hear from you that service is as it should be. Thanks to all for their patience."

UPDATE Tuesday September 5: I haven't seen any intermittent outages at all since Friday, so that problem looks solved now.

"So we have put a new solution in place overnight (Friday Aug 31).

"We installed 4 new state of the art fiber nodes on Mayne and Galiano. What this allows us to do is to utilize specially ordered tuneable transmitters. The 3 Mayne nodes now transmit at different wavelengths, and then recombine in the Pender Headend. No more phasing and colliding with each other.

"These are not parts that we just have lying around and had to be ordered specifically for this application, this is why the solution took as long as it did. We also had to prove out that we had no other choice. Once a permanent solution is put in place with the delivery method, we will reclaim the transmitters as they are extremely expensive and overkill. Your Ping Plotter looks pretty good since the work was completed. Now, any issues will just be regular noise which we can usually track down."

13 comments:

  1. Thanks for your detailed explanation of the problems.

    ReplyDelete
  2. Yes, a big thank you to Chris for getting the facts on the situation.

    ReplyDelete
  3. Thanks Chris, appreciate it. I've called Shaw countless times, but can never get any sort of information out of them. I wish they had made this information available to their front-line support!

    ReplyDelete
  4. Since this is the second time the cable has been severed, perhaps we need to find out who is responsible and look at ways to make them pay for the damage. Are there indicators on shore so people know not to drop anchor near the cables?

    ReplyDelete
    Replies
    1. It would be my guess that Shaw would love to find who did this - if they don't know.

      Delete
  5. Thank you Chris, that is a clear and concise article. I called Shaw again yesterday (Sunday) to see if there was more information, ETA on a fix, etc, but their tech support desk did not relate my modem "flapping" to the fibre problem, instead the local installers will replace my modem. Oh well. Fingers crossed that the microwave link will work and work soon.

    ReplyDelete
  6. Shaw could have had a microwave link in place within days if not hours if they had had the will. We are talking about about a company with very deep pockets. & we are all still being billed for non existant services.

    Shameful

    ReplyDelete
    Replies
    1. Although I'm not often one to be an apologist for large corporations, I think we are lucky to have Shaw serving our island. When I moved here all we had was telephone dial-up and it was sloooooow. When Cablelan came in and brought in broadband it was great, although it was plagued with a lot of problems. Since Shaw has taken it over they have steadily improved things.

      The amount of business Mayne Island represents is probably equal to a city block in Vancouver with a fraction of the infrastructure cost. I doubt if they are investing here for immediate return on their money, I would imagine they are looking at a longer term investment. I have no idea how expensive or easy it is to put in a microwave link to fill in until the fibre is replaced but I suspect it's not nearly as simple or as easy to do as the commenter above casually claims.

      Shaw could pull out of Mayne tomorrow if they were focused on immediate returns on their money and then we'd be all back on dial-up. There are a lot of businesses and community resources on Mayne Island that are dependant on the decent broadband speed we now have and I'm pleased Shaw is committed to remedying the situation.

      By the way, if anyone thinks they are being unfairly billed for service they haven't received, my experience when the internet was out in the past was that Shaw was willing to refund my disputed amount without much pushing from me.

      Delete
  7. The day the cable went down, Shaw left a phone message saying that they were aware of the problem and would be dealing with it ASAP .
    The next day they left another message that said they would not charge us for the time it was out.
    I hadn't even called them about it. That is good PR!

    ReplyDelete
  8. A couple of years ago a Shaw installer told me that Shaw brought in over $30,000.00 in fees each month from customers on Mayne.

    I don't think they are hurting....

    And my internet is still crappy, they keep promising to send a tech - they have missed four appointments so far. So colour me unimpresssed.

    ReplyDelete
  9. Chris ... well done ... I am not on Shaw however find find your diligence and willingness to share your findings generous. Also, what a great role for MayneNews Blog ... nice work Jim T.
    Toby

    ReplyDelete
  10. Still experiencing problems on Galiano in the last few days.

    ReplyDelete