Liberate your Cross Connects! The power of public utilities in network interconnection.
Recently, there have been some debates on LinkedIn¹ ² about certain data centers jacking up cross connect fees upon their colocation customers and telecom carriers being threatened or kicked out of the data center for selling telecommunications services to customers in other competing data centers.
These kinds of situations are becoming more common, as certain data center operators, when pressed by competition, are behaving erratically and abuse their market position by either threatening or exploiting their customers’ dependence on their tenancy. Unfortunately, regardless of how many times we’re dismayed by these situations, there is often little a tenant can do when a data center decides to act in this fashion.
Customers of colocation have no rights.
Fundamentally speaking, colocation customers often have no guaranteed rights or legal protections—they’re not leasehold tenants.
When a data center threatens a colocation tenant with non-renewal of service, the customer is often left with the following choices: (i) give into the data center’s exploitive demands or unequal power equation in favor of the data center, (ii) get kicked out from the data center, or (iii) litigate in the court of the law, with little to no guarantees of success.
Further, when the colocation customer happens to be a network provider (such as a telecom carrier or an ISP), and the data center attempts to coerce that carrier into an anti-competitive conduct
Carriers make revenues by selling telecommunications services to other businesses who are also taking space in data centers.
Not only is the data center threatening the carrier with eviction in this case, it is also attempting to attack the carrier’s contracts with its telecommunications clients, threatening the very revenue from services the carrier provides at the property as a tenant.
For carriers, this is the worst form of landlord harassment imaginable.
While coercive conduct against carriers to hurt competition is unlawful under federal anti-trust laws, it sadly occurs repeatedly enough in the industry and is costly to prove that it’s often referred to as “anti-tethering.”
Why are customers so vulnerable and exploitable by certain bad actors in the colocation industry?
Let’s face it, nobody likes moving. It’s a lot of work and costs a lot. The truth is, we only move our businesses when the benefits outweigh the risks and costs, because otherwise, frequent moving of a mission-critical IT operation such as colocation can be destructive to our businesses, resulting in untold amounts of revenue losses.
Imagine if your landlord is harassing you because you happen to be competing in some business or are doing something he does not like in some way. That’s how bad this is, except that in the data center, you’re not a leasehold tenant, so you have no relief or consumer protection under the law.
And that is if you’re able to move your servers and network equipment in the first place.
Often times, for many businesses, moving out of a data center can be even more cost-prohibitive than giving into the data center’s exploitative price hikes or demands, or is simply not possible at all.
Wires are hard to move.
The No. 1 reason as to why colocation consumers are exploited by bad actors is: connectivity.
Whereas servers and IT hardware are physical objects which can move, network interconnections
Moving cross connects for network interconnection requires cooperation by both the consumer and its other party at the opposite end of a cross connect, such as a cloud provider like Amazon Web Services (AWS) or a telecom carrier. Unfortunately, often times, the other party can’t cooperate, because they’re physically deployed only in that one data center, making it cost-prohibitive or impossible for the connection to move at all.
Unlike electric power, every network interconnection brings so much intrinsic value to a business that it’s hard to deliver them as a universal service.
If the party you’re trying to connect to is only deployed at a dominant and exploitative data center, that’s often the only place you’ll be able to cross connect with them. This gives that one data center an unequal leverage, where it will now have the power to exploit the situation and significantly raise the price of its cross connects and force its tenants into accepting deals that are less favorable.
How did we come to this? Why are network interconnections so vulnerable?
So why do we physically “fix” our cross connects to data centers and make them so vulnerable to shenanigans such as these?
You may have heard of the term “carrier-neutral data centers (CNDCs).”
CNDCs became popular because the data center offered itself as a neutral meeting ground for different businesses and competing telecom carriers to come in and connect between each other. Because the data center itself is not offering or forcing its own brand of connectivity services, it allows its customers to establish network connections at a neutral meeting place, without dictating upon these customers on whom they can connect with, hence “carrier-neutral.”
Unfortunately, CNDCs have become victims of their own success.
You see, success begets competition. Imitation is often the best form of flattery—when one CNDC is wildly successful in a metro market, new data center developers will start coming in to compete with that incumbent CNDC. This resulted in a few CNDCs becoming overly protective of their incumbent dominance, abusing their leverage to coerce customers against moving to a competing CNDC. When that failed to produce enough results, the incumbent CNDCs would then turn their ire on telecom carriers, intimidating them into limiting connectivity services offered at other competing CNDCs.
So, yea, it’s regretful to see that “carrier-neutral” data centers of today are no longer neutral as they once used to be.
Make no mistake though, there are many amazing and fantastic CNDCs today who are extremely pleasurable to work with and are staying true to their core mission. For the avoidance of the doubt, in this article, we’re only talking about bad actors causing problems in the market.
Free the cross connects!
It is starting to become apparent that bad conduct by certain CNDCs has ruined the colocation market’s trust for making vital network interconnections. By locating our cross connects to a single dominant CNDC, we make our businesses vulnerable to unilateral and exploitive price increases during tough economic times, and for carriers, we’re often living under the constant threat of termination of service if we don’t play ball to the abusive CNDC’s liking.
It is high time that we free our cross connects from this nightmare, once and for all.
Mobility is life.
Tactical mobility
If your critical connectivity depends on immovable cross connects that are physically fixed to a single CNDC or building, then you have lost all tactical mobility and have zero chance of survival when that CNDC becomes exploitive and turns its ire on you.
To achieve tactical mobility for network interconnections, it’s time to move mission-critical cross connects (cross connects that you can’t run your digital business without) out of the exploitative CNDC and over to an even more neutral meeting ground, as certain CNDCs have violated the sanctity and trust of “carrier-neutrality” in recent years and therefore can’t be trusted anymore.
The interconnection war spills onto the streets.
Do you know what is so amazing about a cross connect? It’s a passive and dumb piece of wire that costs almost nothing to run.
It takes no power, it generates no heat, it costs almost nothing, and yet, we constantly let our businesses get ruined over cross connects when a CNDC we rely on decides to play greedy.
Since cross connects are essentially a series of tubes, there is a perfect place to run them. When was the last time you’ve been outside and saw a telephone pole with wires on them? Yea, I thought so.
The public right-of-way is an unbelievably amazing place for network interconnections. It’s the public’s easement or land, run by the government for the benefit and convenience of all—it can’t discriminate and is fundamentally prohibited from becoming greedy. In fact, the public’s rights-of-way are used not just for us to drive our cars and walk or bike ourselves over, but they’re often critical transmission arteries for our modern society. Everything we rely on, from electric power to water and sanitation, and telecommunications depend on the public rights-of-way—without these public ways, we simply can’t function as a society.
So, given that our businesses are becoming ever more reliant on having assured access to dependable and cost-effective network interconnections, it’s only apropos that we start moving some of our most vulnerable and critical cross connects out to public ways.
So now we’re all becoming our own dark fiber providers?
By locating the cross connects out to the public streets, you’ve “untethered” your interconnections from becoming fixed to a single CNDC, and therefore, you’ve achieved tactical mobility in network interconnections. The data center to you now is just a place with reliable power, cooling and physical space—it’s just real-estate to you at this point, because by running the majority of your network interconnections outdoors, you’ve effectively become your own ‘mini’ dark fiber provider to cross connect with others.
By installing your most important and vulnerable cross connects outdoors, if an exploitative CNDC begins to threaten your cross connects or fiber cable, you can simply extend your fiber to another competing data center nearby to power your equipment and you’re done.
Because the points of physical connections for your important and critical cross connects are now spliced out at the streets, the exploitative CNDC has suddenly lost all leverage and has nothing on you—the emperor now has no clothes.
Every data center to you is now nothing but an “extension” from where your cross connect splice points are located in the streets.
But you see, becoming even a very small dark fiber provider is no easy task. This often means that you’ll have to become a licensed CLEC, or seek services from another licensed CLEC to host your network out in the streets. But that’s the easy part—aerial utility poles, underground conduits and manholes are often very limited in space to accommodate new cable owners, never mind splice points for your cross connects. Moreover, many owners of conduits will regularly use every excuse under the sun to prevent new tenants from coming in, as they’ll often see new entrants as a competitive threat to their business. Dysfunction is often the name of the game when it comes to running fibers outdoors. 🤣
An open telecom rebellion against exploitive cross connects is in progress in Boston, Massachusetts
Boston is known for many dramatic things—we’re known for the defiant Tea Party protest against British rule from our nation’s founding days, and fast-forward to today: we’re known for our unique blend of politics and the Big Dig, and we’re also apparently quite serious about sports.
Indeed, if there is one word to describe Boston’s naturally sporting culture, I would say “defiance” could be one way to describe it—and, when it comes to our individual freedoms and self-determination, well, this is America, and nobody tells us what to do.
This defiant culture has now fully propagated into Boston’s internet infrastructure, and there is now an open rebellion by telecommunications carriers against exploitative data center cross connect practices—we’ve had enough, and the rent is too damn high. And as the state that blew through $15 billion putting highways underground, we love digging expensive trenches, quite literally speaking.
The public utility enters the fray.
City of Somerville, Massachusetts is a municipality located immediately above Boston proper. The city’s motto is “Municipal Freedom Gives National Strength” and when it comes to technology, it became the first municipality in Massachusetts to establish greater competition in cable TV & internet services in 1997³.
In 2021, the city welcomed the construction of New England’s first underground utility system dedicated to high-scale network interconnections, revolutionizing internet peering, interconnected data center communities and new transatlantic cable developments.
Called the Hub Express System (HEX) and built by TOWARDEX as the lead company of the joint trench, the underground conduit system has turned Somerville’s Inner Belt Road into a massive “Meet-Me Street”, where every telecom manhole is part of an interconnected cross connection fabric, and the entire utility is an open access system, licensing out duct space to everybody for $1.54/foot per year.
The HEX system is operated as a common carrier, a public utility system for fiber attachments under the 1996 Telecommunications Act. Under the jurisdiction of the Massachusetts Department of Telecommunications and Cable, the system owner cannot discriminate, intimidate tenants, or terminate services without complying with the state approved procedure⁴.
As a public utility system, HEX is dutifully bound by law to provide access, regardless of the competition.
Indeed, Section 1 of the Project Charter of the Hub Express System, the founding document establishing the system, defiantly and openly declares war against exploitative cross connect practices⁵:
“we therefore agree to embark upon a lawful undertaking to operate a new multi-cable, multi-conduit fiber optic transmission system dedicated to promoting interconnection freedom, accessibility, network neutrality, digital equality, freedom of enterprise and freedom of expression for all.”
All tenants of the HEX system can easily license conduit space to run new cross connect cables between themselves, but to make things even easier, the system operator is also emplacing 1728-strands fiber cables and network of splice cases connecting all of the HEX manholes and participating nearby utilities.
If a network wishes to request dark fiber capacity (by 12-strands ribbons) from one manhole to another, there is no monthly fee to use those fibers, only an annual membership to cover plant costs & maintenance⁶. No recurring cross connect fees! 😂
Perhaps the best part about the HEX system, however, isn’t that it’s only for use by carriers.
It's the fact that carriers can monetize their fibers in the system by selling retail connectivity to enterprise customers at the streets. This lets customers, even those who aren't telecom carriers, cross connect with others in manholes, which greatly lowers the carrier's risk of being taken advantage of by CNDCs who threaten carriers' colocation licenses for providing services at a competitor’s facility.
In the HEX system, CLECs are able to “sponsor” or “host” a non-CLEC’s network in the system, as long as the CLEC is taking legal ownership and responsibility for the fiber optic cable that is being used by their end customer. In fact, a few carriers are now offering “managed fiber network” service in the HEX system, with an entire cable sold as a private dark fiber service to their retail enterprise customers.
Because the HEX system is so cheap to use and cannot dictate or regulate services offered by carriers over fiber cables they own and manage, a non-CLEC company who is looking to purchase 288-strands of bulk dark fiber without paying cross connects for example, can do so in the HEX system—they would simply contract with a CLEC who is willing to provide this managed turn-key cable service.
The complex relationship between data centers and HEX.
Competing data centers in Boston and new data center developers are viewing HEX through a strategic lens:
Connecting the data center property onto the HEX system promptly unleashes complete and full ecosystem of network providers, as every carrier can instantly pull in their backbone cable on the cheap, without paying exorbitant fees or doing new construction.
If a data center had troubles convincing a carrier to build fiber into their facility, those troubles are now gone, and the data center had to pay little or nothing for that privilege.
But at the same time, the presence of HEX also means that the data center’s colocation customers can now become their own dark fiber provider and run their interconnections out in the streets without being subject to cross connect fees.
Because the customer is pulling in their own backbone cable into the data center through HEX, the only reasonable defense the data center has against this maneuver would be to charge punitive “cable entrance fees” to put an end to the practice—for example, a data center could consider charging a very high $6,000 to $10,000 per month for a customer to bring in a fiber optic cable from the streets.
However, cable entrance fees pose significant risks for the data center operator.
First, by charging exorbitant fees for cable entries, the data center is effectively choking out its own total available fiber capacity into the building, slowly turning itself into an island.
An alternative strategy for the data center could involve pricing discrimination. Instead of charging cable entry fees for carriers, they could charge a judicious sum only to non-carrier retail customers as a means of discouraging this practice.
But perhaps what presents the biggest danger to cable entrance fees is the HEX system’s Universal Open Access Lateral rule⁷:
Every utility lateral connection owned by HEX to any property must be constructed under the same joint-use rule and be openly licensed to everybody at the same cheap price, without any discrimination.
What does this mean? This means that every property with a lateral connection to the HEX system now has equal access to the same ecosystem of fiber carriers working in the streets.
It doesn’t take a rocket scientist to soon realize that another competing data center provider connected to the HEX system could now steal the entire colocation customer by simply waiving cable entrance fees. Losing an entire colocation customer revenue over a cable entrance fee is an unacceptable lose-lose situation for most data center operators, even for bad actors.
And also, it’s not just data centers competing against each other that is at issue here.
Even commercial properties (offices, industrial warehouses, etc.) connected to the HEX system are now a potential competitive Meet-Me Room threat.
Savvy radio guys and small ISPs have no problems setting up DC plants and turning any office lease/sub-lease into their own network hub overnight by pulling fiber from the HEX system, eliminating cross connect fees left and right and everywhere.
In fact, an industrial warehouse on Inner Belt Road was recently quoted $900/month for a long-haul 100G link to 60 Hudson in New York—like, what the hell, it’s now becoming cheaper to buy backbone connectivity at a warehouse property than at a professional data center… The market has truly gone mad.
Expanding upon the European IX model of keeping colocation operators honest.
So, we now arrive at a paradox:
HEX effectively nullifies the monopoly connectivity leverage held by incumbent CNDCs, and allows new competing data centers (who also charge for cross connects) to virtually become network-dense overnight and openly challenge the incumbent CNDCs in the market.
But at the same time, HEX also creates cross connect revenue risks for these competitive data centers being benefited by the system, because the system allows everyone, including non-CLEC companies to take advantage of the utility network to do their own cross connects out in the streets. It’s a godsend blessing and a poison pill at the same time for the data center operator.
But here is the thing, and this is why data centers (even bad actors in the industry) often view the HEX system from a strategic perspective:
When you’re focused on printing money, never lose a colocation customer. Losing a decent-sized tenant hits you financially harder than would-be lost cross connect fees.
Now, customers have more options for network interconnection:
They can (i) host high-volume, critical, and most important cross connects out in the streets with no recurring fees; and (ii) purchase a monthly in-building cross connect at a data center for any circuits where ease-of-ordering and flexibility are more desirable than saving costs.
Because customers host their critically important cross connects out in the streets, cable entry into a data center is now becoming a mandatory requirement in any competitive colocation bid.
The HEX system has effectively “de-fanged” the data centers’ leverage over network interconnections in Boston, encouraging them to instead play nice and compete fairly in every new colocation opportunity, or risk losing out to a competitor.
A representative from a preeminent data center company had quipped:
“Look, we’re not going to fight the utility out in the street; we welcome the rocketing connectivity and fiber density at our facility and are laser-focused on becoming a dependable partner for our customers to grow and succeed with us.”
Sound and effective regulatory frameworks can help restore trust in the marketplace.
What’s happening with cross connects and data center connectivity right now in Boston is both a remarkable market success story and a cautionary tale at the same time for the colocation industry.
The first major battle of utilities and data centers had been won and lost, and both sides knew which is which.
This is a story demonstrating the true power of public utilities in network interconnections, and it’s also a testament to the Commonwealth of Massachusetts’ sound and informed regulatory policy for encouraging competitive telecommunications, while carefully balancing the policy to ensure consumer protections.
Instead of regulating entry of a new common carrier through overly burdensome requirements, Massachusetts Department of Public Utilities held that “there are benefits inherent in a competitive marketplace that encourage greater levels of economic efficiency and fairness than does a regulated monopoly environment” (Mass. D.P.U. 93-98, Order (“Common Carrier Regulatory Treatment Order”) at 11) and further held that the Department’s “tariff regulations, statutory requirements, and consumer complaint resolution process, are sufficient to ensure not only that rates are just and reasonable but that there is adequate consumer protection for interexchange, competitive access, and AOS services, absent the regulation of entry into these markets.” (Id. at 12).
Together with the Telecommunications Act of 1996, Massachusetts’ framework, which eliminated market entry regulations, is what allowed HEX to be constructed, enabling new entrants like TOWARDEX to register as a common carrier and join forces with other common carriers to innovate and promote a more competitive and fair marketplace for consumers of data center services.
It’s one thing to piss off and threaten carriers, but when you threaten their livelihoods enough that they start forming a public utility to fight back against exploitative cross connects, an all-out telecom rebellion is what results.
This whole thing may now just become the story of how Boston is joining the Tier-1 cities’ club in North America for global internet traffic, built on free cross connects! 😆
Citations
1. LinkedIn Post: https://www.linkedin.com/posts/jvanoppen_we-got-a-random-price-increase-for-cross-activity-7216509223403159552-TR_u
4. 220 CMR 45.03(3)(a) through (e)
7. https://infrastructure.twdx.net/hex/Open_Access_Lateral.pdf
Large Enterprise Account Executive @ Crown Castle | Building Resilient Connectivity
1yJames, very well written and thought out. Thank you so much for the education here!
I grow people, revenue and profit.
1yThank you for sharing your market knowledge & solution
Founder, Board Member and CEO @ Lightboard | Telecom Infrastructure and Real Estate Expert
1yThanks for the mention! Another great addition to the tool box supporting network optimization! And having a front row seat in Boston you can see its great work. 👍
Yuku Fujikawa makes a great point regarding the facility (Carrier Hotel, Colocation, Data Center) being run by a carrier (or service provider) in that the landlord is competing directly with its' tenants. This has evolved as the operators of facilities have begun to sell internet, transit, transport, etc, so in essence they are really MSP's that run facilities. The better option, if you can find it, is to license space with a truly neutral custodian. The MSP's that run facilities either in a conscious or unconscious state draw business and revenue away from their tenants instead of fully driving business to their tenants which in turn creates the highest probability for success and a long term tenancy...and if you don't charge MRC on cross connects you raise that level of success.
Technical MBA w/AI Blockchain and Crypto, Co-Founder Equinix, Syntropy, ConsoleConnect and DrPeering Labs
1yRelative to the cost of the next best alternative (fiber or circuits outside the data center), cross connects are a bargain. I calculated once that successful data centers could charge thousands of dollars per month and it would still not cause networks to leave. Behold the gravitational power of colo critical mass.