SQL Server 2014 – Troubleshooting Incorrect Job Next Run Time

jobsscheduled-taskssql serversql server 2014sql-server-agent

I got an issue regarding job stats. So I have a job with the following schedule: Occurs every day every 5 minute(s) between 12:00:00 AM and 11:59:59 PM. Schedule will be used starting on 12/13/2017.

in 11:27 AM 2019-10-20 I ran two scripts to get last_run_time and next_run_time, below are the scripts and their corresponding results:

USE msdb
GO
SELECT J.Name                     AS 'Job Name'
         ,'Job Enabled' =        
          CASE J.Enabled
                 WHEN 1 THEN 'Yes'
                 WHEN 0 THEN 'No'
          END
         ,STUFF(
          STUFF(CAST([active_start_date] AS VARCHAR(8)), 5, 0, '-')
          , 8, 0, '-')                   AS 'Job Schedule Start Date'
         ,STUFF(
          STUFF(CAST([active_end_date] AS VARCHAR(8)), 5, 0, '-')
          , 8, 0, '-')                   AS 'Job Schedule End Date'
         ,'Job Frequency' =
          CASE S.freq_type
                 WHEN 1 THEN 'Once'
                 WHEN 4 THEN 'Daily'
                 WHEN 8 THEN 'Weekly'
                 WHEN 16 THEN 'Monthly'
                 WHEN 32 THEN 'Monthly relative'
                 WHEN 64 THEN 'When SQLServer Agent starts'
          END
         ,'Last Run Date' = CONVERT(DATETIME, RTRIM(LASTRUN.run_date) + ' '
          + STUFF(STUFF(REPLACE(STR(RTRIM(LASTRUN.run_time), 6, 0),
          ' ', '0'), 3, 0, ':'), 6, 0, ':'))
         , 'Last Run Status' =
          CASE
                 WHEN LASTRUN.run_status = 0 THEN 'Failed'
                 WHEN LASTRUN.run_status = 1 THEN 'Succeeded'
                 WHEN LASTRUN.run_status = 2 THEN 'Retry'
                 WHEN LASTRUN.run_status = 3 THEN 'Cancelled'
                 ELSE 'Unknown'
          END                           
         ,'Last Run Message' = LASTRUN.message
FROM dbo.sysjobs J
LEFT OUTER JOIN dbo.sysjobschedules JS
       ON J.job_id = JS.job_id
LEFT OUTER JOIN dbo.sysschedules S
       ON JS.schedule_id = S.schedule_id
LEFT OUTER JOIN (SELECT
              job_id
          ,MAX(run_duration) AS RUN_DURATION
       FROM dbo.sysjobhistory
       GROUP BY job_id) MAXDUR
       ON J.job_id = MAXDUR.job_id
LEFT OUTER JOIN (SELECT
              J1.job_id
          ,J1.RUN_DURATION
          ,J1.run_date
          ,J1.run_time
          ,J1.message
          ,J1.run_status
       FROM dbo.sysjobhistory J1
       WHERE instance_id = (SELECT
                     MAX(instance_id)
              FROM dbo.sysjobhistory J2
              WHERE J2.job_id = J1.job_id)) LASTRUN
       ON J.job_id = LASTRUN.job_id
ORDER BY 'Job Name'

use msdb
go
Select [job name] = jb.name 
    , [schedule name] = sc.name
    , [next run date] = js.next_run_date
    , [next run time] = js.next_run_time
From dbo.sysjobs as jb
Inner Join dbo.sysjobschedules as js on js.job_id = jb.job_id
Inner Join dbo.sysschedules as sc on js.schedule_id = sc.schedule_id

So, according to the pictures next_run_date should be 11:30 AM instead of 11:15 AM. Have you ever faced with this kind of phenomenon? Could you please help me understand this?

Best Answer

The docs on sysjobschedules reveal that the data inside refreshes every 20 minutes.

NOTE: The sysjobschedules table refreshes every 20 minutes, which may affect the values returned by the sp_help_jobschedule stored procedure.

Which is why your next_run_date & next_run_time columns could show out of date data.

For a much longer answer & deep dive. Check out this answer by Aaron Bertrand

He talks about using dbo.sysjobactivity.

You could use the next_scheduled_run_date, which is a datetime column from this dmv.

Example query:

SELECT sj.Name, 
next_scheduled_run_date
FROM msdb.dbo.sysjobs sj
JOIN msdb.dbo.sysjobactivity sja
ON sj.job_id = sja.job_id
WHERE session_id = (
    SELECT MAX(session_id) FROM msdb.dbo.sysjobactivity);

With the max(session) used because the agent starts a new session each time it restarts.

This results in

Name                            next_scheduled_run_date
LSAlert_My_machine              2019-10-20 11:34:00.000
LSRestore_My_machine_Test42     2019-10-20 11:45:00.000
LSCopy_My_machine_Test42        2019-10-20 11:45:00.000
syspolicy_purge_history         2019-10-21 02:00:00.000

SSMS

When tracing what SSMS is doing to get the job info when calling job activity monitor. the last_run_date & last_run_time are taken from the xp_sqlagent_enum_jobs procedure, in a cursor wise manner.

You could get the job info & place a cursor over it by using this procedure.

An example of getting the info for one job:

DECLARE @job_id uniqueidentifier
DECLARE @job_owner varchar(255)
SELECT @job_owner = SUSER_SNAME()

SELECT @job_id=job_id FROM msdb..sysjobs 
WHERE name = 'LSAlert_My_machine'
    EXECUTE master.dbo.xp_sqlagent_enum_jobs 1, @job_owner, @job_id;

2nd until 5th columns returned by the procedure:

Last Run Date   Last Run Time   Next Run Date   Next Run Time
20191020        114400          20191020        114600

If you wish to run the procedure for all jobs:

DECLARE @job_id uniqueidentifier, @job_name varchar(255)
DECLARE @job_owner varchar(255)
SELECT @job_owner = SUSER_SNAME()
DECLARE @xp_results TABLE(Job_name varchar(255),Job_Id varbinary(16),last_run_date varchar(10),last_run_time varchar(10),next_run_date varchar(10), next_run_time varchar(10), next_run_schedule_id int, requested_to_run bit,request_source int, request_source_id int, running bit, current_step int, current_retry_attempt int, [State] int)

DECLARE C CURSOR FAST_FORWARD READ_ONLY FOR 
SELECT [name],job_id FROM msdb..sysjobs 
OPEN C
FETCH NEXT FROM C INTO @job_name,@job_id
WHILE @@FETCH_STATUS = 0
BEGIN

INSERT INTO @xp_results(Job_Id,last_run_date ,last_run_time ,next_run_date , next_run_time , next_run_schedule_id , requested_to_run ,request_source , request_source_id , running , current_step , current_retry_attempt , [State] )
    EXECUTE master.dbo.xp_sqlagent_enum_jobs 1, @job_owner, @job_id;

UPDATE @xp_results
SET Job_name = @job_name
WHERE Job_Id = @job_id;

FETCH NEXT FROM C INTO @job_name,@job_id
END
CLOSE C
DEALLOCATE C
SELECT Job_name,last_run_date,last_run_time,next_run_date,next_run_time 
FROM @xp_results;

Example output

Job_name                        last_run_date   last_run_time   next_run_date   next_run_time
LSAlert_My_machine              20191020        115800          20191020        120000
LSCopy_My_machine_Test42        20191020        114500          20191020        120000
LSRestore_My_machine_Test42     20191020        114500          20191020        120000

Short Answer

It looks like the data in msdb.dbo.sysjobschedules is updated by a background thread in SQL Agent, identified as SQLAgent - Schedule Saver, every 20 minutes (or less frequently, if xp_sqlagent_notify has not been called and no jobs have run in the meantime).

For more accurate information, look at next_scheduled_run_date in msdb.dbo.sysjobactivity. This is updated in real-time any time a job is changed or a job has run. As an added bonus, the sysjobactivity stores the data the right way (as a datetime column), making it a lot easier to work with than those stupid INTs.

That's the short answer:

It could be up to 20 minutes before sysjobschedules reflects the truth; however, sysjobactivity will always be up to date. If you want a lot more details about this, or how I figured it out...

Long Answer

If you care to follow the rabbit for a moment, when you call sp_add_jobschedule, this chain of events is set into motion:

msdb.dbo.sp_add_jobschedule == calls ==> msdb.dbo.sp_add_schedule
                                         msdb.dbo.sp_attach_schedule

msdb.dbo.sp_attach_schedule == calls ==> msdb.dbo.sp_sqlagent_notify

msdb.dbo.sp_sqlagent_notify == calls ==> msdb.dbo.xp_sqlagent_notify

Now, we can't chase the rabbit any further, because we can't really peek into what xp_sqlagent_notify does. But I think we can presume that this extended procedure interacts with the Agent service and tells it that there has been a change to this specific job and schedule. By running a server-side trace we can see that, immediately, the following dynamic SQL is called by SQL Agent:

exec sp_executesql N'DECLARE @nextScheduledRunDate DATETIME 
  SET @nextScheduledRunDate = msdb.dbo.agent_datetime(@P1, @P2) 
  UPDATE msdb.dbo.sysjobactivity 
    SET next_scheduled_run_date = @nextScheduledRunDate 
    WHERE session_id = @P3 AND job_id = @P4',
N'@P1 int,@P2 int,@P3 int,@P4 uniqueidentifier',
20120819,181600,5,'36924B24-9706-4FD7-8B3A-1F9F0BECB52C'

It seems that sysjobactivity is updated immediately, and sysjobschedules is only updated on a schedule. If we change the new schedule to be once a day, e.g.

@freq_type=4, 
@freq_interval=1, 
@freq_subday_type=1, 
@freq_subday_interval=0, 
@freq_relative_interval=0, 
@freq_recurrence_factor=1,

We still see the immediate update to sysjobactivity as above, and then another update after the job is finished. Various updates come from background and other threads within SQL Agent, e.g.:

SQLAgent - Job Manager
SQLAgent - Update job activity
SQLAgent - Job invocation engine
SQLAgent - Schedule Saver

A background thread (the "Schedule Saver" thread) eventually comes around and updates sysjobschedules; from my initial investigation it appears this is every 20 minutes, and only happens if xp_sqlagent_notify has been called due to a change made to a job since the last time it ran (I did not perform any further testing to see what happens if one job has been changed and another has been run, if the "Schedule Saver" thread updates both - I suspect it must, but will leave that as an exercise to the reader).

I am not sure if the 20-minute cycle is offset from when SQL Agent starts, or from midnight, or from something machine-specific. On two different instances on the same physical server, the "Schedule Saver" thread updated sysjobschedules, on both instances, at almost the exact same time - 18:31:37 & 18:51:37 on one, and 18:31:39 & 18:51:39 on the other. I did not start SQL Server Agent at the same time on these servers, but there is a remote possibility that the start times happened to be 20 minutes offset. I doubt it, but I don't have time right now to confirm by restarting Agent on one of them and waiting for more updates to happen.

I know who did it, and when it happened, because I placed a trigger there and captured it, in case I couldn't find it in the trace, or I inadvertently filtered it out.

CREATE TABLE dbo.JobAudit
(
  [action] CHAR(1),
  [table] CHAR(1), 
  hostname SYSNAME NOT NULL DEFAULT HOST_NAME(), 
  appname SYSNAME  NOT NULL DEFAULT PROGRAM_NAME(),
  dt DATETIME2     NOT NULL DEFAULT SYSDATETIME()
);

CREATE TRIGGER dbo.schedule1 ON dbo.sysjobactivity FOR INSERT
AS
  INSERT dbo.JobAudit([action], [table] SELECT 'I', 'A';
GO
CREATE TRIGGER dbo.schedule2 ON dbo.sysjobactivity FOR UPDATE
AS
  INSERT dbo.JobAudit([action], [table] SELECT 'U', 'A';
GO
CREATE TRIGGER dbo.schedule3 ON dbo.sysjobschedules FOR INSERT
AS
  INSERT dbo.JobAudit([action], [table] SELECT 'I', 'S';
GO
CREATE TRIGGER dbo.schedule4 ON dbo.sysjobschedules FOR UPDATE
AS
  INSERT dbo.JobAudit([action], [table] SELECT 'U', 'S';
GO

That said, it is not hard to catch with a standard trace, this one even comes through as non-dynamic DML:

UPDATE msdb.dbo.sysjobschedules 
  SET next_run_date = 20120817, 
      next_run_time = 20000 
 WHERE (job_id = 0xB87B329BFBF7BA40B30D9B27E0B120DE 
 and schedule_id = 8)

If you want to run a more filtered trace to track this behavior over time (e.g. persisting through SQL Agent restarts instead of on-demand), you can run one that has appname = 'SQLAgent - Schedule Saver'...

So I think that if you want to know the next run time immediately, look at sysjobactivity, not sysjobschedules. This table is directly updated by Agent or its background threads ("Update job activity", "Job Manager" and "Job invocation engine") as activity happens or as it is notified by xp_sqlagent_notify.

Be aware, though, that it is very easy to muck up either table - since there are no protections against deleting data from these tables. (So if you decided to clean up, for example, you can easily remove all the rows for that job from the activity table.) In this case I'm not exactly sure how SQL Server Agent gets or saves the next run date. Perhaps worthy of more investigation at a later date when I have some free time...

Sql-server – SQL join query to show rows with non-existent rows in one table

Thank you for SQLfiddle and sample data! I wish more questions started this way.

If you want all members regardless of whether they have an entry for that date, you want a LEFT OUTER JOIN. You were very close with this version however a little trick with outer joins is that if you add a filter to the outer table in the WHERE clause, you turn an outer join to an inner join, because it will exclude any rows that are NULL on that side (because it doesn't know if NULL would match the filter or not).

I modified the first query to get a row for every member:

SELECT Members.Member_ID
      ,Time_Entry.Date_Start
      ,Time_Entry.Hours_Actual
      ,Time_Entry.Hours_Bill
FROM dbo.Members
  LEFT OUTER JOIN dbo.Time_Entry
--^^^^ changed from FULL to LEFT
  ON Members.Member_ID = Time_Entry.Member_ID
  AND Time_Entry.Date_Start = '20131110';
--^^^ changed from WHERE to AND

I'll leave it as an exercise for the reader to take it from there and add the other columns, formatting, COALESCE etc.

Some other notes:

please always use the schema prefix when creating and referencing objects
please always use a length when converting to varchar etc.
stay away from ambiguous, regional date formats like mm-dd-yyyy

consider using aliases to make your queries easier to read. E.g. the above could be re-written as:

SELECT m.Member_ID
  ,t.Date_Start
  ,t.Hours_Actual
  ,t.Hours_Bill
FROM dbo.Members AS m
LEFT OUTER JOIN dbo.Time_Entry AS t
ON m.Member_ID = t.Member_ID
AND t.Date_Start = '20131110';

... a lot tidier, IMHO, as long as you use sensible aliases.

Best Answer

Related Solutions

SQL Agent Next Run Date – How and When SQL Agent Updates next_run_date/next_run_time

Short Answer

Long Answer

Sql-server – SQL join query to show rows with non-existent rows in one table

Related Question