Technical SEO Explained

Technical SEO is the least discussed and most consequential part of the discipline, for one reason: its failures are silent.

Write a bad article and you can see it is bad. Ship a noindex header on a template, or a canonical pointing at the wrong URL, and everything looks perfect

• the page renders, the content is good, nothing errors. It simply never appears in search, and nothing tells you.

This is the layer that determines whether any of your other work is visible at all. It is also, encouragingly, the most finite: a checklist you can complete, rather than an ongoing effort.

Can a crawler reach your pages?

Everything starts here. Nothing else matters if the answer is no.

robots.txt: the file that can quietly delete your site

robots.txt tells crawlers where they may go. It is four lines of text and one of the easiest ways to remove a site from search entirely:

User-agent: *
Disallow: /

That says "nobody crawl anything". It appears in production more often than you would think, usually copied from a staging config.

What you normally want:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

One subtlety worth understanding: Disallow prevents crawling, not indexing. A blocked URL can still appear in results if other sites link to it - the engine knows it exists and cannot see what is on it. To keep a page out of the index, let it be crawled and serve noindex. Blocking it in robots.txt actually prevents the engine from seeing the noindex you added.

Sitemaps: discovery without luck

A sitemap lists your URLs and when they last changed. It does not guarantee indexing; it makes discovery reliable rather than dependent on link structure.

Generate it - a hand-maintained sitemap is a stale sitemap. In Next.js it is one file:

export default function sitemap(): MetadataRoute.Sitemap {
  return getAllPosts().map((post) => ({
    url: `${base}/blog/${post.slug}`,
    lastModified: new Date(post.date),
  }));
}

Because it derives from the same data the pages use, a new post appears automatically and an unpublished one never leaks.

Internal links are the real crawl path

Crawlers follow links. A page reachable only by typing its URL is effectively invisible, sitemap or not.

The practical failure: a blog index that paginates with JavaScript-only controls, so page two never gets crawled and everything on it disappears from search.

Can it understand and index the page?

Reached is not the same as understood.

Rendering: the JavaScript problem

Google renders JavaScript, and that fact gets quoted as though the problem were solved. The detail matters: rendering happens in a second pass, queued and resourced separately. Content in the initial HTML is processed immediately; content requiring JS execution waits, sometimes days, sometimes not at all. Other search engines and most social preview crawlers do far less.

Which is why the rendering strategy decides your SEO ceiling:

• Static (SSG) - HTML at build time. Best case; every crawler sees everything.
• Server-rendered (SSR) - HTML per request. Equally visible.
• Client-rendered (CSR) - an empty shell plus JS. Worst case; you are relying on the second pass.

For content that must rank, static or server-rendered is not an optimisation, it is the requirement.

The quickest check: view source (not DevTools' Elements panel, which shows the DOM after JS). If your article text is not in the source, crawlers are not reliably seeing it either.

Canonicals and duplicates

The same content reachable at several URLs splits its own signals and competes with itself. Common causes: www and non-www, http and https, trailing slashes, tracking parameters, and print views.

A canonical tag names the authoritative version:

<link rel="canonical" href="https://example.com/blog/technical-seo-explained" />

Set it on every page, including the canonical one - self-referencing canonicals are correct and prevent parameterised copies competing.

The classic bug is a canonical hardcoded in a shared layout, so every page declares the homepage as its canonical. The site looks fine and only the homepage gets indexed.

Metadata and structured data

Each page needs a unique <title> and description. Duplicates across a site are a strong signal that the pages are near-identical.

Structured data (JSON-LD) states explicitly what a page is - an article, a product, an FAQ - rather than leaving it to inference:

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Technical SEO Explained",
  "datePublished": "2026-03-20",
  "author": { "@type": "Person", "name": "Nawal Kattel" }
}

It does not raise rankings directly. It makes rich results - author, date, ratings, FAQ accordions - possible, and those raise click-through.

Speed, stability, and the rest

The final layer is about whether the page is pleasant enough to keep someone there.

Core Web Vitals, briefly

Three measurements Google uses:

• LCP - when the largest element appears. Target under 2.5s. Usually an image or a webfont.
• INP - responsiveness to interaction. Target under 200ms. Usually long-running JavaScript on the main thread.
• CLS - how much the layout jumps. Target under 0.1. Usually images without dimensions, or ads and banners injected above content.

They are a modest ranking factor and a large user-experience one. CLS in particular is worth fixing for its own sake: content shifting under a reader's finger is infuriating.

The most common wins are unexciting - size your images, set explicit width/height, self-host fonts with font-display: swap, and ship less JavaScript.

The silent-failure checklist

Worth running before assuming a ranking problem is about content:

• Is the page in the index? Search site:example.com/your-page.
• Does view-source contain the content?
• Is there an accidental noindex header or meta tag?
• Does the canonical point at this page?
• Is it reachable by clicking from the homepage?
• Is it in the sitemap, and does Search Console report the sitemap as read?
• Does it load acceptably on a mid-range phone on 4G?

Search Console's URL Inspection answers most of these directly, including how Google rendered the page. It is the single most useful free tool in SEO and most people never open it.

Key takeaways

• Technical SEO fails silently. The page looks perfect and never appears.
• Disallow blocks crawling, not indexing - and blocking a page prevents engines from seeing its noindex.
• Generate the sitemap from your data, so it cannot go stale.
• Content in the initial HTML is indexed reliably; JS-rendered content waits for a second pass that may not come.
• Set a self-referencing canonical on every page - and check it is not hardcoded to the homepage.
• Structured data does not rank you higher; it makes rich results possible, which raises clicks.
• Use Search Console's URL Inspection before theorising about rankings.

FAQ

My page is not indexed. What is the first thing to check?

URL Inspection in Search Console. It tells you whether Google has crawled it, whether it was indexed, what canonical it chose, and how it rendered - which usually identifies the cause immediately.

Does Google really run JavaScript?

Yes, in a separate, later, resource-limited pass. Rely on it and you accept a delay and a risk. Other crawlers - Bing, social previews, LLM crawlers - do substantially less.

Do I need AMP?

No. It has been deprecated as a requirement for top stories, and fast ordinary pages achieve the same thing without the constraints.

How important is HTTPS?

Non-negotiable. It is a ranking signal, and browsers warn users away from sites without it.

Should every page be indexed?

No. Tag pages, filtered listings, and paginated archives often should not be - they dilute the index with near-duplicates. Use noindex deliberately.

How often should I audit?

Check Search Console monthly and run a full audit after any framework upgrade, redesign, or URL change. Those three are when silent breakage is introduced.

Conclusion

Technical SEO is unglamorous and finite. There is a defined set of things that can be wrong, most are checkable in an afternoon, and once fixed they stay fixed until you change something structural.

That combination makes it the highest-return work in SEO - not because it will lift you above a better page, but because it decides whether you are in the race at all. Content and links compete; crawlability is a gate.

Read more

What Is SEO? for the wider picture, On-Page SEO vs Off-Page SEO for what sits above this layer, and How Google Search Works for what the crawler does with the pages it collects.