Improve Crawling and
Indexing with
User-Friendly
Log File Insights
Slideshare.Net/Stevenvvessum
@STEVENVVESSUM
Steven van Vessum
Conductor
Steven van Vessum
Director of Organic Marketing at Conductor
Previously:
Co-founder & VP of Community at ContentKing
Title Slide Option
3
Real-Time SEO Auditing & Monitoring
About
#brightonSEO
4
Real-Time SEO Auditing & Monitoring
About
#brightonSEO
5
Technical SEO is the
Foundation of Successful
Organic Marketing
Technical SEO
Technical
Ensure discoverability
Content
Address customer needs
Insights
Optimize performance
Action
Move faster
About
#brightonSEO
Let’s be honest…
We’re all having issues
getting content indexed.
Possible
reasons…
● The web is growing
faster than ever.
● The rise of AI
generated content.
Photo by the
awesome Lily
Ray from
Amsive Digital
Possible
reasons…
Crawling, rendering,
indexing, storing and
serving content is
expensive.
Possible
reasons…
● Google’s has been
raising the “content
quality bar” over the
last few years.
● Helpful Content
Update in August
2022.
Tough shit? 💩
The Issues We See
at Companies
The issue we
see at
companies
They just ship an X
amount of content
pieces a month.
The issue we
see at
companies
They don’t reflect on
the work they’ve done.
The issue we
see at
companies
They don’t understand
and can’t explain
obvious search engine
behavior.
How Companies
Get Better
Enablement Companies need
easy access to
log file insights.
Thinking
critically and
getting
answers
● Has Google crawled
these pages yet?
● Has Google tried
crawling pages
while they had
issues?
● How often do my
pages get recrawled
on average?
● Has Google
reindexed the pages
we updated?
Thinking
critically and
getting
answers
With these
answers, they
can….
● Improve time to
crawl
● Improve time to
index
● Improve time to
rank
● Improve time to
traffic
● SEO forecast
With these
answers, they
can….
By providing easy access
to log file insights….
You’ll make SEO
accessible.
You’ll make SEO fun.
You’ll get buy-in from
management.
Log files
What are
log files?
● Text files containing
records of all the
requests a server
has received from
both humans and
crawlers.
● Its responses to
these requests.
Why Are Logs
Important to SEO?
Logs show crawlers’
true behavior
Log File Analysis As
We Know It
Log file analysis
as we know it Inefficient and time
consuming.
Often only performed
once a year.
Log file analysis
as we know it
Insights are kept
within silos.
Log file analysis
as we know it
I get it though.
But we NEED
to do better
(and we can)
Our Dream
● Log data is
available in
real-time.
● Insights are
automatically
distilled.
It would be
amazing if….
It would be
amazing if….
● Made easily
accessible for
web and
content teams.
Getting Your Hands
on Logs
Is….tricky
● Access logs aren’t
there (anymore).
● PII concerns.
Is….tricky
● Office politics.
● Slow processes.
Enter CDN logs
● Real-Time
● Plug ‘n Play
Connecting to
Content Inventory
Connect log
insights to
content
inventory
● Coupled to
content
● Results in
enlightenment
Improving Crawling
& Indexing
New Pages How long does it take
for search engines to
crawl new pages?
New Pages Can you speed up
time to crawl
with better content
promotion?
New Pages Can you speed up time
to crawl with better
internal linking?
New Pages Is there a correlation
between
crawl frequency and
internal links?
New Pages Is there a correlation
between
crawl frequency and
internal links?
Move away from
manually parsing raw
logs to automatically
extracted insights.
Updated Pages Has Google already
picked up my
improvements?
Updated Pages How soon does Google
refresh pages after
making updates?
Updated Pages Is there increased crawl
activity after making
changes?
A timeline with
changes and whether
they were seen by
search engines.
Robots.txt Has Google recrawled
the new robots.txt
directives?
Robots.txt Has Google tried
requesting the
robots.txt while it had a
Disallow: /
Learn what search
engines know.
XML Sitemap Has Google recrawled
my XML sitemap after it
was updated?
XML Sitemap How frequently does
Google crawl my oldest
XML sitemaps?
Learn what content
search engines know
about.
Recap
Logs are important
to SEO
Lots of companies
underperform without insights
from logs
Distilling actionable insights
from logs is time-consuming
and hard
Making insights easily
accessible across teams
is hard
Getting your hands
on logs is hard
Connecting
insights to content
inventory is hard
Enter CDN logs.
(With ContentKing)
Learn how
your work moves
the needle
And keep sharpening
your blade.
Shout!
Questions, comments, ideas…
FREE 45-day Trial
Experience Real-Time Log File Analysis yourself.
Improve your SEO game in under 20 seconds.
contentkingapp.com/brightonseo

BrightonSEO October 2022 - Log File Analysis - Steven van Vessum.pdf