The Daily Parker

Politics, Weather, Photography, and the Dog

Fifth month in a row over 50

This is my 55th post this month, and the fifth month in a row in which I've posted over 50 times. That brings my 12-month total to 581, the third record in a row and the fifth record this year. I guess Covid-19 has been good for something.

Here's what I'm reading today:

I'm excited to add a notch on the Brews and Choos project in a few hours. Check back tomorrow.

Two stark comparisons

First: the difference between how Garmin handled a global outage that lasted 5 days, and how SendGrid managed one that lasted 5 hours. SendGrid handles billions of emails per day, including for Microsoft and other massive companies. So SendGrid going down didn't inconvenience a few athletes and pilots; it crippled Fortune-500 companies' marketing departments. (And it delayed a scheduled release on my own team.)

Within about an hour of their outage, SendGrid created an incident response page to which they posted updates every half-hour. They clearly stated what was going on and how they were trying to fix it, even as they were discovering for themselves what had happened:

Contrast that with Garmin, who still haven't really explained what happened or why it took so long to resolve the outage. (They finally declared their remediation "complete" yesterday, almost 6 days after the outage started.)

Second: the difference between how Germany (and other rich countries) have handled Covid-19 and how we have. Josh Marshall looked into the numbers:

The head of Germany’s equivalent of the CDC told reporters today that he’s “very concerned” about the rising case numbers in the country and accuses Germans of becoming “negligent” in their adherence to mitigation measures. He has good reason to be concerned.

Today Germany reported 638 new cases of COVID. That comes out to .76 cases per 100,000 residents. Let’s round that up to 1 new case per 100,000 for good measure. (The need for round numbers will become clear.)

Today New York State had 3 cases per 100,000. So Germany is concerned by 1/3 the number of cases as we have in New York state, a state which is probably controlling the disease as well as any other state in the country.

Florida today had 43 cases per 100,000. Florida’s outbreak is more than forty-three times the size relative to population.

To state the point baldly, Germany is very concerned about a rise in cases that would still be dramatically better than any other part of the United States. They’re ramping up border restrictions to get things back under control and chiding the population to redouble its collective efforts.

[S]eeing this as, "well, look, everyone’s having problems." Or "we’re not the only ones having new outbreaks" or "we can’t go back to shutdowns…" ... only captures how Americans are having a hard time grappling with just how many universes away we are from what is happening in other countries which are comparably affluent, industrialized and able to mount an effective response.

We’re failing that badly.

Anthony Fauci was on BBC just now, struggling not to call the president out on his criminal negligence. Only 98 days until we can vote the bastard out.

Another outage

Even as Garmin picks up the pieces from what they now admit was a massive ransomware attack, bulk email provider SendGrid has gone down spectacularly.

I use SendGrid, as does my company, for status emails and such.

Here's my problem, though: I have a code update to put out that specifically targets a bug in SendGrid's .NET library that they claim to have fixed. My automated build pipelines won't release new code unless all the unit tests pass. Right now, the SendGrid tests fail sporadically, and at least one fails every time the tests run.

Ah, technology.

How to pass secrets into your custom log target

Today I finally solved a problem that has nagged me for months: Given a .NET Core 3.1 Web API, NLog, and a custom log target for NLog, how could I pass secrets into the log target using Azure Key Vault?

The last bit required me to add Azure Key Vault support to the InnerDrive.Logging NuGet package, which I did back in February. The KeyVaultSecretProvider class allows you to retrieve secrets from a specified Azure Key Vault. (I'm in the process of updating it to handle keys as well.) Before that, you'd have to put your secrets in configuration files, which anyone on your development team or who has access to your Git repository can see. For example, to use our SendGridTarget class, you would have to put this in your nLog.config file:

<extensions>
	<add assembly="NLog.Web.AspNetCore"/>
	<add assembly="InnerDrive.Logging"/>
</extensions>

<targets async="false">
	<target xsi:type="SendGridTarget"
	        name="sendgrid"
		apiKey="{supposedly secret key}"
		applicationName="My Cool App"
	        from="service@contoso.org"
	        to="admin@contoso.org"
	/>
</targets>

That is...suboptimal. Instead, you want to use a class that implements ISecretProvider and inject it into the SendGridTarget class through this constructor:

/// <summary>
/// Creates a new instance of <see cref="SendGridTarget"/> with
/// a specified <see cref="ISendGridSender"/> implementation.
/// </summary>
/// <param name="sender">The <see cref="ISendGridSender"/> to use</param>
/// <param name="secretProvider">The <see cref="IConfiguration"/> to use</param>
/// <param name="apiKey">The API key to use</param>
public SendGridTarget(
	ISendGridSender sender, 
	ISecretProvider secretProvider = null, 
	string apiKey = null)
{
	IncludeEventProperties = true;
	if (null != secretProvider) Configuration = secretProvider;
	if (!string.IsNullOrWhiteSpace(apiKey)) ApiKey = apiKey;
	_sender = sender;
}

And now I'm going to save you about 30 hours of research and frustration. To get NLog to inject the secret provider into the target, I added this method to Startup.cs:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env, IComponentContext container)
{
	// Setup code elided
	ConfigureLogging(container);
}

public static void ConfigureLogging(IComponentContext container)
{
	var defaultConstructor = ConfigurationItemFactory.Default.CreateInstance;
	var secretProvider = container.Resolve<ISecretProvider>();
	ConfigurationItemFactory.Default.CreateInstance = type =>
	{
		if (type == typeof(SendGridTarget))
		{
			var sendGridSender = container.Resolve<ISendGridSender>();
			return new SendGridTarget(sendGridSender, secretProvider);
		}

		return defaultConstructor(type);
	};
	LogManager.Configuration = LogManager.Configuration.Reload();
	NLogBuilder.ConfigureNLog(LogManager.Configuration);
}

(Note that the ISecretProvider and ISendGridSender classes need to be registered with your dependency-injection framework.)

Today was a good day.

Garmin offline

Four days after I switched from Fitbit to Garmin, all of Garmin's online services have gone offline:

The problems with those services also mean that a range of features can't be used on Garmin's own devices: it is not possible to create new routes to go running or cycling, for instance, or to share those activities on services like Strava once they are completed.

The devices themselves continue to work as normal with the data they do have, however, meaning that any data collected during the outage will be safe.

Garmin wrote on its official Twitter pages that the problems were also affecting its call centres, leaving users unable to get in touch through calls or online messages.

"We are currently experiencing an outage that affects Garmin Connect, and as a result, the Garmin Connect website and mobile app are down at this time," it wrote.

"This outage also affects our call centres, and we are currently unable to receive any calls, emails or online chats. We are working to resolve this issue as quickly as possible and apologise for this inconvenience."

This is not what Garmin customers want to see:

The error message on the Garmin Connect website suggests the problem is with their Cloudflare equipment:

Update: Based on Garmin employee's social-media posts, ZDNet now reports that the company experienced a catastrophic ransomware attack, most likely a new strain of WastedLocker. Fortunately, my Venu can hold 200 hours of data. So as long as they get it back up within a week or so, I shouldn't lose anything—unless the ransomware attack already destroyed my data from this past week.

Did someone call "lunch?"

I think today is Tuesday, the first day of my 10th week working from home. That would make today...March 80th? April 49th? Who knows.

It is, however, just past lunchtime, and today I had shawarma and mixed news:

Earlier, I mentioned that the state's unemployment office accidentally revealed thousands of records in an own goal. Turns out, Deloitte Consulting did the work, so I am no longer surprised. Note to anyone who needs software written: don't hire a big consulting firm. They don't attract the best developers because they use manager-driven development patterns that irritate the hell out of anyone with talent.

Domestic terrorism in Michigan

Charlie Pierce, noting that "[p]eople with firearms forced the civil government of the state of Michigan to shut itself down," wants to know in what sense this isn't terrorism. In other fun weekend stories:

And it's pouring, and will continue to do so for several more hours.

The plan is to have no plan

So believes NYU media professor Jay Rosen about how President Trump will try to win this fall:

The plan is to have no plan, to let daily deaths between one and three thousand become a normal thing, and then to create massive confusion about who is responsible— by telling the governors they’re in charge without doing what only the federal government can do, by fighting with the press when it shows up to be briefed, by fixing blame for the virus on China or some other foreign element, and by “flooding the zone with shit,” Steve Bannon’s phrase for overwhelming the system with disinformation, distraction, and denial, which boosts what economists call “search costs” for reliable intelligence.

Stated another way, the plan is to default on public problem solving, and then prevent the public from understanding the consequences of that default. ... The manufacture of confusion is just the ruins of Trump’s personality meeting the powers of the presidency. There is no genius there, only a damaged human being playing havoc with our lives.

In other fun stories:

Oh, and 151 years ago today, the Union Pacific and Central Pacific railroads completed the Transcontinental Railroad.

What's a Wednesday again?

Remember slow news days? Me neither.

  • Republican legislators and business owners have pushed back on Illinois Governor JB Pritzker's plan to re-open the economy, preferring instead to force their employees into unsafe situations so they can return to making money.
  • Professional dilettante Jared Kushner's leadership in getting a bunch of kids to organize mask distribution went about as well as one might predict.
  • More reasonable people simply see how it means we're going to be in this a while.
  • California has sued Uber and Lyft for violating AB5, claiming the two ride-sharing companies “gain an unfair and unlawful competitive advantage by inappropriately classifying massive numbers of California drivers as independent contractors,” according to California Attorney General Xavier Becerra.
  • Assuming states were allowed to go bankrupt, Crain's Steven Strahler believes an Illinois bankruptcy might not be what anyone actually wants.
  • Illinois' $560m shortfall in gasoline taxes right now has put transit projects at risk.
  • The BBC tries to help the rest of the world understand why the US has a backlash against face masks, as does NBC.
  • If you take New York, New Jersey, and Connecticut out of the equation, the number of Covid-19 cases continues to rise in the US.
  • Bottled water sales have gone up 57% year-over-year, so Consumer Reports wants to know why people are paying so much for someone else's tap water? Especially since bottlers often don't pay their water bills while residents are getting their water shut off.
  • Anyone remember that it's the 20th anniversary of the ILOVEYOU virus?

And finally, a cute diner in Toronto where I had breakfast last June has moved to delivery service during the lockdown. Too bad they can't deliver to Chicago.

Gosh, where to begin?

Happy May Day! Or m'aidez? Hard to know for sure right now. The weather in Chicago is sunny and almost the right temperature, and I have had some remarkable productivity at work this week, so in that respect I'm pretty happy.

But I woke up this morning to the news that Ravinia has cancelled its entire 2020 season, including a performance of Bernstein's White House Cantata that featured my group, the Apollo Chorus of Chicago. This is the first time Ravinia has done so since 1935.

If only that were everything.

First, via Josh Marshall, former Obama Administration disaster-preparedness expert Jeremy Konydndyk lays out the facts about our plateau (60,000 excess weekly deaths) and how the Trump Administration continues to do nothing to help us slow Covid-19 deaths.

Next, all of this:

But some good news:

Finally, while alarming in its own right, the record water levels in Lake Michigan (4 months in a row now) have exposed some historic shipwrecks.