About the Author
Richard Lingsch
Enterprise Infrastructure Strategy, Nubius Solutions
Richard has spent 30+ years in infrastructure, hosting, cloud, and application delivery, from IT consulting at Deloitte to co-founding eApps Hosting, where he led the shift from Domino and Java/Tomcat hosting to Xen, KVM, and enterprise OpenNebula. He works with midmarket companies reassessing VMware and virtualization economics, helping them segment workloads and migrate only where the business case holds.
Hybrid connectivity projects have a predictable shape. Week one, someone draws a diagram with a cloud on one side, a building on the other, and a line between them. Week twelve, the line exists, the tunnel is up, BGP has converged, and nothing works properly because DNS resolves differently depending on which side of the link you are standing on, the MTU is wrong so large payloads vanish silently, and the firewall team has a rule set that predates the project by four years and nobody wants to touch it.
The connectivity itself is a solved problem. Google Cloud offers well-documented options and they work. What is not solved, and what consumes the schedule, is everything the connection exposes: address space that was never designed to be routable, name resolution that was never designed to be split, security controls that were never designed to be traversed, and applications that were never designed to be 20 milliseconds further away from their dependencies.
This is a practitioner’s view of building hybrid connectivity from an on-premises estate into GCP, focused on the decisions that are expensive to reverse.
Choose the Transport Before You Choose Anything Else
There are three realistic transports and they sit on a spectrum of cost, latency predictability, and lead time.
Cloud VPN over the public internet is the fastest to stand up. High availability configurations give you a documented uptime posture using redundant tunnels across separate interfaces, and for many workloads that is entirely sufficient. The limitation is not availability. It is throughput ceiling per tunnel and latency variance. Your traffic crosses the public internet, which means your latency is whatever the internet decides it is that afternoon. For asynchronous replication, batch transfers, and management traffic, this is fine. For a chatty application tier making synchronous calls to an on-premises database, it is not.
Dedicated Interconnect gives you physical circuits into a Google colocation facility, with predictable latency and high throughput. Traffic is not encrypted by default, so environments that require encryption can use MACsec or HA VPN over Cloud Interconnect. The cost is lead time measured in weeks or months, a colocation presence, and cross-connect ordering. Plan it early or do not plan it at all.Partner Interconnect sits in between. You connect through a service provider that already has a presence in the Google facility, which removes the colocation requirement and shortens lead time considerably. Bandwidth options are more granular. For most mid-sized estates this is the pragmatic landing spot.
The decision that actually matters is not which one you pick. It is whether you build one or two. A single Interconnect circuit is a single point of failure with a physical dependency on a fibre path you do not control. If the hybrid link carries production traffic, it needs redundancy across separate edge availability domains, and ideally a VPN as a backup path with a deliberately less preferred BGP configuration so failover is automatic and predictable rather than a manual scramble.
Address Space Is the Constraint You Cannot Engineer Around Later
Address-space overlap is one of the problems that can derail a hybrid project late in the process.Your on-premises estate has an address plan that grew organically. Somewhere in it is a 10.0.0.0/16 that someone allocated in 2014. Your GCP VPC needs a CIDR range. If they overlap, straightforward routing between the environments is no longer possible. Re-addressing or address translation becomes necessary, adding complexity to both the design and future troubleshooting.Audit the full on-premises address space before you allocate a single subnet in GCP. Not the documented space. The actual space, including the lab environments, the acquired company’s network that got bridged in three years ago, and the management network for the storage array. Then allocate GCP ranges from a block that is probably unused and reserve enough headroom that you are not back here in eighteen months.
GCP’s VPC model helps here in one specific way that is worth understanding. Subnets are regional, not zonal, and the VPC itself is global. This means you do not need a separate VPC per region and you do not need to peer regions together, which removes an entire class of address planning complexity that other providers impose. Use it. A single well-planned VPC with regional subnets and a clean CIDR allocation is far easier to route to from on-premises than a mesh of peered networks.
Also plan for the ranges you do not control. Google-managed services, private service access ranges, and GKE pod and service CIDRs all consume address space, and some of them need to be routable from on-premises if your workloads talk to them. Allocate for them up front.
Routing: Cloud Router, BGP, and the Advertisements You Forgot
Cloud Router handles dynamic routing over BGP, and it works well once you understand what it advertises by default and what it does not.
By default, Cloud Router advertises the subnets of the VPC it lives in. It does not advertise the ranges used for private service access, it does not advertise GKE pod ranges in every configuration, and it does not advertise anything from peered VPCs. If your on-premises hosts need to reach a Cloud SQL instance over private service access, you need a custom route advertisement. This is a two-line configuration change that teams spend days diagnosing because the symptom is a timeout with no useful error.
VPC Network Peering is non-transitive, so on-premises connectivity does not automatically extend into a peered VPC. Reaching those networks requires deliberate route exchange, including Cloud Router custom advertisements for the peered subnet ranges and the appropriate custom-route import and export settings on the peering. Decide on that topology before you build.
On the return path, verify what on-premises is advertising to Cloud Router. Overly broad advertisements from the on-premises side can create routing conflicts. Overly narrow ones mean return traffic finds a default route out to the internet and gets dropped by an egress firewall you had forgotten about.
Route priority is your failover mechanism. If you have Interconnect as primary and VPN as backup, the BGP MED values determine which path is preferred. Test the failover deliberately by shutting the primary session, not by waiting for an outage to test it for you.
MTU: The Failure Mode That Looks Like Everything Else
This deserves its own section because it is responsible for an unreasonable share of hybrid networking incidents, and the symptom never points at the cause.
Packets over a VPN tunnel carry encapsulation overhead. The effective MTU on the path is lower than the standard 1500 bytes that both your on-premises hosts and your cloud instances assume. When an application sends a full-size packet with the do-not-fragment bit set, and it hits a link that cannot carry it, the correct behaviour is an ICMP fragmentation-needed message back to the sender, which then reduces its segment size. This is Path MTU Discovery, and it works beautifully right up until a firewall somewhere in the path drops ICMP because a security policy from 2016 said ICMP is dangerous.
What you get then is a black hole. Small packets work perfectly. TCP handshakes complete. Pings succeed. SSH connects and then hangs the moment output exceeds a certain size. HTTP requests work and HTTP responses with a large body time out. Database connections establish and then stall on the first substantial result set. Every one of these symptoms sends engineers looking at the application, the database, and the load balancer, because the network is clearly fine, look, ping works.
For Cloud VPN, verify that the peer VPN gateway is configured for Google’s documented MTU requirements and that ICMP fragmentation-needed or Packet Too Big messages are permitted where required so Path MTU Discovery can function. Google Cloud VPN already performs TCP MSS clamping, so pay particular attention to non-TCP traffic, the peer gateway configuration, and any firewalls along the path.
Test it explicitly. Send progressively larger packets with fragmentation disabled and find the exact size where they stop arriving. Do this before go-live, not during the incident.
DNS: Two Sources of Truth, One Set of Clients
Hybrid DNS is where the project stops being a networking exercise and starts being an integration exercise.
You have on-premises resolvers that know about internal zones. You have Cloud DNS with private zones that know about GCP resources. Neither knows about the other, and your workloads on both sides need to resolve names from both.
Cloud DNS forwarding zones send queries for specific on-premises domains to your internal resolvers. Inbound server policies create resolver endpoints inside the VPC that on-premises DNS servers can forward to. You need both directions configured, and you need the firewall rules to permit the resolver traffic on both TCP and UDP port 53. UDP-only DNS rules are a classic omission, and they produce failures only for large responses, which means they work for months and then break the day someone adds a record.
The DNS traffic itself must route over the hybrid link, which means the resolver ranges need to be advertised and reachable. Verify resolution from inside the actual workload runtime. Resolution from a jump host proves nothing about what a GKE pod with its own DNS layer will do.
Watch for split-horizon conflicts. If the same domain name resolves to different addresses depending on which resolver answers, you have a race condition that will surface as intermittent, unreproducible failures.
Latency Is an Architecture Decision, Not a Network Problem
Once the link is up, the physics do not negotiate. If your application tier moves to GCP and its database stays on-premises, every query now carries the round-trip time of the hybrid path. A request that makes forty sequential database calls has just added forty times the round-trip latency to its response time.
This is not something the network team can fix. It is an application architecture question, and it needs to be answered during planning by identifying which components are chatty with each other and refusing to split them across the link. Components that talk constantly belong on the same side. Components that exchange batches or events can be split.
Problems arise when the hybrid link is treated as a transparent extension of the LAN. It is still a WAN, and applications need to be designed accordingly.This is precisely the category of problem we describe in our breakdown of cloud migration risks, where latency implications get discovered after the workloads have already moved.
Cost Modelling the Egress Nobody Counted
Interconnect has a port charge and an egress charge. VPN has a tunnel charge and an egress charge. The port and tunnel charges are predictable. The egress is not, because nobody knows in advance how much data will cross the link once applications start using it.
Replication traffic, backup traffic, log shipping, monitoring telemetry, and container image transfers can all cross the hybrid boundary. Model both the direction and the volume against real observed traffic. Google generally charges for data transferred out through the connection, while inbound data transfer is generally not charged by Google. The pattern of a well-understood base cost expanding into an unpredictable variable cost is the same one we examine across cloud infrastructure in general in our analysis of why cloud bills become unpredictable, and hybrid egress is one of its most reliable sources.
Operating It After It Works
Monitor BGP session state and route counts, not just tunnel status. A tunnel that is up while the BGP session has withdrawn half its routes is a partial outage that reports as healthy.
Monitor the effective path MTU on a schedule. Firewall changes on either side can silently reintroduce the black hole months after go-live.
Document the failover behaviour and test it quarterly. The value of a redundant path is entirely dependent on it working the day the primary fails, and untested failover paths have a poor record.
Keep the address plan under change control. The moment someone allocates an overlapping range on either side, the routing breaks for everyone.
Where Nubius Fits
Nubius operates across AWS, Azure, GCP, Vultr, and on-premises environments, and hybrid designs are where that breadth matters most. Under Nubius OpsAssist AnyCloud we support cloud, private, and hybrid estates with cost optimisation, security posture assessment, performance monitoring, and the kind of advanced troubleshooting that hybrid links generate.
The load balancers, VPN endpoints, and core Linux services sitting on both ends of the link fall under Nubius Managed AppOps, which covers HAProxy, GCP load balancing, operating system services, and the monitoring and alerting around them.
If you are planning a hybrid build and want the address plan, routing design, and MTU behaviour validated before the circuit order goes in, our cloud migration and custom projects team can help. You can also talk to an engineer directly.