Wednesday, March 05, 2025

CVS and Publix Coupon Clippers as Userscript

Several months ago, CVS added more security to their site to prevented my bookmarklet solution from injecting the clipper code to clip all the coupons. So, I've revised the CVS clipper as well as the Publix clipper to now be Userscripts. The advantage of this is that the CVS clipper now works again and they both now automatically add a button on the page to perform the "Clip All" action which is easier to use (no need to find the bookmarklet bookmark). The downside is that it requires a browser extension to be installed, and then the clipper script needs to be installed in the extension.

Please go to https://pothoven.net/CvsClipper.html or https://pothoven.net/PublixClipper.html for installation instructions, and please drop me a comment here if you try it out to let me know how it works for you.

Friday, July 07, 2023

Migrating away from Angular flex-layout

In a blog post from October 22, 2022 the Angular team announced

Layout has made significant advancements since Angular’s early days. Based on advancements to native layout solutions and removal of support for IE11, the Angular team will stop publishing new releases of the experimental @angular/flex-layout library starting in v15.

They then proceeded to layout migration alternatives.

  • CSS Flexbox
  • CSS Grid
  • TailwindCSS

This has resulted in considerable consternation for developers.  I do not wish to restate what others have already said about the problem and will instead simply point you to a good article on the topic entitled, Why the Deprecation of Flex-Layouts Is Concerning for Angular Developers. What I intend for this article is provide a simple and cleaner migration alternative off of the @angular/flex-layout package. The migration is actually a step back in time to utilize a set of CSS Flexbox classes defined for AngularJS and I need to give a shout-out to my team member, Pam Tingiris, for remembering and formulating this solution.

I am a solutions architect over a team of developers. During the past few years we have developed a number of Angular applications that are farily large and involved. When we started we were obviously using an older version of Angular without the @angular/flex-layout package. We then followed the Angular team's recommendation to utilize the @angular/flex-layout package so we migrated to it. While the migration was time consuming and tedious just due to the volume of changes to make, it was actually fairly straight-forward and was simply the inverse of what I'm recommended for you today. Fortunately, in our code repository we were able to go back and find the old AngularJS flex layout classes that we had previously migrated away from and deleted. I'm included it at the end of this post.

The migration is mostly a straight conversion from the @angular/flex-layout injected DOM stylings to CSS classes. For example, let's look at the initial example provided on the @angular/flex-layout GitHub page with an addition of a fxFlex div child:

<div fxLayout="row" fxLayoutAlign="space-between">
  <div fxFlex></div>
</div>

With the CSS classes, that simply becomes:

<div class="layout-row layout-align-space-between">
  <div class="flex"></div>
</div>

As you can see, it's a pretty straight-forward process. When you review the CSS at the bottom of the page, you will see there are classes for most every layout variation you might use. For example, fxLayoutAlign="space-between center" becomes class="layout-align-space-between-center" and fxFlex="50" becomes class="flex-50".

For the most part, we standardized our layout gaps to 1rem (example fxLayoutGap="1rem") so we added a single layout-gap class at the end of the original CSS file (.layout-gap { gap: 1rem }), but for the few occasions when we needed different spacing in a component we'd just add a custom class in the component's CSS such as .layout-gap-3rem { gap: 3rem }.

Now, you might be thinking to yourself, that's all well and good for basic flex layout directives, but what about this part of flex-layout?

The real power of Flex Layout, however, is its responsive engine. The Responsive API enables developers to easily specify different layouts, sizing, visibilities for different viewport sizes and display devices.

Here's where it's a little harder, but still not terrible. We'll utilize the Material CDK layout package to create a BreakpointObserver service.

You inject this service into any component that needs media queries for responsive displays.

  constructor(
    public breakpoints: BreakpointsService,
  ) { }

Now, if you have a page that is laid out in rows, but you want it to switch to columns for a phone display, you might have something like:

<div fxLayout="row" fxLayout.xs="column" fxLayoutAlign="space-between">
</div>

Utilizing the breakpoints, this will become:

<div class="layout layout-align-space-between"> [ngClass]="{'layout-column' : breakpoints.screen('xs')}">
</div>

Admittedly not as nice as the concise fxLayout.xs, but it does the job. The built in breakpoint names of the BreakpointObserver like XSmall don't correspond to the flex-layout names like xs, so we named them to match. Our BreakpointsService looks like:

/********************************************************************************
 * BreakpointsService (a BreakpointObserver)
 *
 * Implementation of the Material CDK BreakpointObserver
 * see https://material.angular.io/cdk/layout/overview#breakpointobserver
 * This defines a set of viewport breakpoints (media queries) that allow us
 * to react to changes in viewport sizes for a responsive UI.
 ********************************************************************************/

import { Injectable, OnDestroy } from '@angular/core';
import { BreakpointObserver, Breakpoints} from '@angular/cdk/layout';
import { takeUntil } from 'rxjs/operators';
import { BehaviorSubject, Subject } from 'rxjs';

@Injectable({
  providedIn: 'root'
})
export class BreakpointsService implements OnDestroy {
  destroyed = new Subject<void>();

  mediaQueryMap = new Map([
    ['xs', Breakpoints.XSmall ],
    ['sm', Breakpoints.Small],
    ['md', Breakpoints.Medium],
    ['lg', Breakpoints.Large],
    ['xl', Breakpoints.XLarge],
    ['gt-xs', '(min-width: 600px)'],
    ['gt-sm', '(min-width: 960px)'],
    ['gt-md', '(min-width: 1280px)'],
  ]);

  protected matchesSubject = new BehaviorSubject([]);
  matches = this.matchesSubject.asObservable();

  constructor(public breakpointObserver: BreakpointObserver) {
    breakpointObserver
      .observe([
        Breakpoints.XSmall,
        Breakpoints.Small,
        Breakpoints.Medium,
        Breakpoints.Large,
        Breakpoints.XLarge,
        '(min-width: 600px)',
        '(min-width: 960px)',
        '(min-width: 1280px)',
      ])
      .pipe(takeUntil(this.destroyed))
      .subscribe(result => {
        const matchedBreakpoints = Object.entries( result.breakpoints ).filter( e => e[1] );
        this.matchesSubject.next(matchedBreakpoints);
      });
  }

  ngOnDestroy() {
    this.destroyed.next();
    this.destroyed.complete();
  }

  /***
   * Screen sizes for responsive styling
   * @param size 'xs', 'gt-xs'
   * */
  screen(size) {
    let screenSize = false;
    const matches = this.matches.subscribe(m => {

      const mq = this.mediaQueryMap.get(size);
      screenSize = m.filter( match => match[0] === mq ).length === 1;

    } );
    return screenSize;
  }

}

And that's it! Now it's just a process of finding all the "fx" directives and any screen size rules, and swapping them out for classes. Once you remove the FlexLayoutModule from your app.module.ts, you'll quickly discover anything you might have missed.

import { FlexLayoutModule } from '@angular/flex-layout';

@NgModule({
  imports: [
    FlexLayoutModule,

In conclusion, we found this migration path to be largely a 1-to-1 replacement. We've seen so many other packages come into favor and then fall out of favor over time (like bootstrap) that we didn't want to go down the route of switching to TailwindCSS just to have it ripped out from under us again in a few years. Hopefully you find this to be a helpful alternative.

Flex Layout CSS classes

/* flex layout classes - instructions https://material.angularjs.org/latest/layout/introduction */

.layout-align, .layout-align-start-stretch {
  justify-content: flex-start;
  align-content: stretch;
  align-items: stretch
}

.layout-align-start, .layout-align-start-center, .layout-align-start-end, .layout-align-start-start, .layout-align-start-stretch {
  justify-content: flex-start
}

.layout-align-center, .layout-align-center-center, .layout-align-center-end, .layout-align-center-start, .layout-align-center-stretch {
  justify-content: center
}

.layout-align-end, .layout-align-end-center, .layout-align-end-end, .layout-align-end-start, .layout-align-end-stretch {
  justify-content: flex-end
}

.layout-align-space-around, .layout-align-space-around-center, .layout-align-space-around-end, .layout-align-space-around-start, .layout-align-space-around-stretch {
  justify-content: space-around
}

.layout-align-space-between, .layout-align-space-between-center, .layout-align-space-between-end, .layout-align-space-between-start, .layout-align-space-between-stretch {
  justify-content: space-between
}

.layout-align-space-evenly, .layout-align-space-evenly-center, .layout-align-space-evenly-end, .layout-align-space-evenly-start, .layout-align-space-evenly-stretch {
  justify-content: space-evenly
}

.layout-align-center-start, .layout-align-end-start, .layout-align-space-around-start, .layout-align-space-between-start, .layout-align-start-start, .layout-align-space-evenly-start {
  align-items: flex-start;
  align-content: flex-start
}

.layout-align-center-center, .layout-align-end-center, .layout-align-space-around-center, .layout-align-space-between-center, .layout-align-start-center, .layout-align-space-evenly-center {
  align-items: center;
  align-content: center;
  max-width: 100%
}

.layout-align-center-center > *, .layout-align-end-center > *, .layout-align-space-around-center > *, .layout-align-space-between-center > *, .layout-align-start-center > *, .layout-align-space-evenly-center > * {
  max-width: 100%;
  box-sizing: border-box
}

.layout-align-center-end, .layout-align-end-end, .layout-align-space-around-end, .layout-align-space-between-end, .layout-align-start-end, .layout-align-space-evenly-end {
  align-items: flex-end;
  align-content: flex-end
}

.layout-align-center-stretch, .layout-align-end-stretch, .layout-align-space-around-stretch, .layout-align-space-between-stretch, .layout-align-start-stretch, .layout-align-space-evenly-stretch {
  align-items: stretch;
  align-content: stretch
}

.flex {
  flex: 1
}

.flex, .flex-grow {
  box-sizing: border-box
}

.flex-grow {
  flex: 1 1 100%
}

.flex-initial {
  flex: 0 1 auto;
  box-sizing: border-box
}

.flex-auto {
  flex: 1 1 auto;
  box-sizing: border-box
}

.flex-none {
  flex: 0 0 auto;
  box-sizing: border-box
}

.flex-noshrink {
  flex: 1 0 auto;
  box-sizing: border-box
}

.flex-nogrow {
  flex: 0 1 auto;
  box-sizing: border-box
}

.flex-0, .layout-row > .flex-0 {
  flex: 1 1 100%;
  max-width: 0;
  max-height: 100%;
  box-sizing: border-box
}

.layout-row > .flex-0 {
  min-width: 0
}

.layout-column > .flex-0 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 0%;
  box-sizing: border-box
}

.flex-5, .layout-row > .flex-5 {
  flex: 1 1 100%;
  max-width: 5%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-5 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 5%;
  box-sizing: border-box
}

.flex-10, .layout-row > .flex-10 {
  flex: 1 1 100%;
  max-width: 10%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-10 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 10%;
  box-sizing: border-box
}

.flex-15, .layout-row > .flex-15 {
  flex: 1 1 100%;
  max-width: 15%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-15 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 15%;
  box-sizing: border-box
}

.flex-20, .layout-row > .flex-20 {
  flex: 1 1 100%;
  max-width: 20%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-20 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 20%;
  box-sizing: border-box
}

.flex-25, .layout-row > .flex-25 {
  flex: 1 1 100%;
  max-width: 25%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-25 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 25%;
  box-sizing: border-box
}

.flex-30, .layout-row > .flex-30 {
  flex: 1 1 100%;
  max-width: 30%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-30 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 30%;
  box-sizing: border-box
}

.flex-35, .layout-row > .flex-35 {
  flex: 1 1 100%;
  max-width: 35%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-35 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 35%;
  box-sizing: border-box
}

.flex-40, .layout-row > .flex-40 {
  flex: 1 1 100%;
  max-width: 40%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-40 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 40%;
  box-sizing: border-box
}

.flex-45, .layout-row > .flex-45 {
  flex: 1 1 100%;
  max-width: 45%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-45 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 45%;
  box-sizing: border-box
}

.flex-50, .layout-row > .flex-50 {
  flex: 1 1 100%;
  max-width: 50%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-50 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 50%;
  box-sizing: border-box
}

.flex-55, .layout-row > .flex-55 {
  flex: 1 1 100%;
  max-width: 55%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-55 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 55%;
  box-sizing: border-box
}

.flex-60, .layout-row > .flex-60 {
  flex: 1 1 100%;
  max-width: 60%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-60 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 60%;
  box-sizing: border-box
}

.flex-65, .layout-row > .flex-65 {
  flex: 1 1 100%;
  max-width: 65%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-65 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 65%;
  box-sizing: border-box
}

.flex-70, .layout-row > .flex-70 {
  flex: 1 1 100%;
  max-width: 70%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-70 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 70%;
  box-sizing: border-box
}

.flex-75, .layout-row > .flex-75 {
  flex: 1 1 100%;
  max-width: 75%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-75 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 75%;
  box-sizing: border-box
}

.flex-80, .layout-row > .flex-80 {
  flex: 1 1 100%;
  max-width: 80%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-80 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 80%;
  box-sizing: border-box
}

.flex-85, .layout-row > .flex-85 {
  flex: 1 1 100%;
  max-width: 85%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-85 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 85%;
  box-sizing: border-box
}

.flex-90, .layout-row > .flex-90 {
  flex: 1 1 100%;
  max-width: 90%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-90 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 90%;
  box-sizing: border-box
}

.flex-95, .layout-row > .flex-95 {
  flex: 1 1 100%;
  max-width: 95%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-column > .flex-95 {
  max-height: 95%
}

.flex-100, .layout-column > .flex-95 {
  flex: 1 1 100%;
  max-width: 100%;
  box-sizing: border-box
}

.flex-100 {
  max-height: 100%
}

.layout-column > .flex-100, .layout-row > .flex-100 {
  flex: 1 1 100%;
  max-width: 100%;
  max-height: 100%;
  box-sizing: border-box
}

.flex-33 {
  max-width: 33.33%
}

.flex-33, .flex-66 {
  flex: 1 1 100%;
  max-height: 100%;
  box-sizing: border-box
}

.flex-66 {
  max-width: 66.66%
}

.layout-row > .flex-33 {
  flex: 1 1 33.33%
}

.layout-row > .flex-66 {
  flex: 1 1 66.66%
}

.layout-column > .flex-33 {
  flex: 1 1 33.33%
}

.layout-column > .flex-66 {
  flex: 1 1 66.66%
}

.layout-row > .flex-33 {
  max-width: 33.33%
}

.layout-row > .flex-33, .layout-row > .flex-66 {
  flex: 1 1 100%;
  max-height: 100%;
  box-sizing: border-box
}

.layout-row > .flex-66 {
  max-width: 66.66%
}

.layout-row > .flex {
  min-width: 0
}

.layout-column > .flex-33 {
  max-height: 33.33%
}

.layout-column > .flex-33, .layout-column > .flex-66 {
  flex: 1 1 100%;
  max-width: 100%;
  box-sizing: border-box
}

.layout-column > .flex-66 {
  max-height: 66.66%
}

.layout-column > .flex {
  min-height: 0
}

.layout, .layout-column, .layout-row {
  box-sizing: border-box;
  display: flex
}

.layout-column {
  flex-direction: column
}

.layout-row {
  flex-direction: row
}

.layout-padding-sm > *, .layout-padding > .flex-sm {
  padding: 4px
}

.layout-padding, .layout-padding-gt-sm, .layout-padding-gt-sm > *, .layout-padding-md, .layout-padding-md > *, .layout-padding > *, .layout-padding > .flex, .layout-padding > .flex-gt-sm, .layout-padding > .flex-md {
  padding: 8px
}

.layout-padding-gt-lg > *, .layout-padding-gt-md > *, .layout-padding-lg > *, .layout-padding > .flex-gt-lg, .layout-padding > .flex-gt-md, .layout-padding > .flex-lg {
  padding: 16px
}

.layout-margin-sm > *, .layout-margin > .flex-sm {
  margin: 4px
}

.layout-margin, .layout-margin-gt-sm, .layout-margin-gt-sm > *, .layout-margin-md, .layout-margin-md > *, .layout-margin > *, .layout-margin > .flex, .layout-margin > .flex-gt-sm, .layout-margin > .flex-md {
  margin: 8px
}

.layout-margin-gt-lg > *, .layout-margin-gt-md > *, .layout-margin-lg > *, .layout-margin > .flex-gt-lg, .layout-margin > .flex-gt-md, .layout-margin > .flex-lg {
  margin: 16px
}

.layout-wrap {
  flex-wrap: wrap
}

.layout-nowrap {
  flex-wrap: nowrap
}

.layout-fill {
  margin: 0;
  width: 100%;
  min-height: 100%;
  height: 100%
}

.layout-gap { gap: 1rem }

Friday, January 06, 2023

Reporting errors to Slack from Java Spring Boot and .NET Framework applications

Watching server logs for errors is a cumbersome way to find problems in server code.  

Back when I was actively doing Ruby on Rails for personal development, I used the "exceptional" gem which would report exceptions to a now defunct online service (exceptional.io) to monitor any errors that occurred in your application.  It was a nice tool, but it had 2 drawbacks for me.  

  1. It was just for Ruby on Rails
  2. It required a 3rd party site to store the exception data.  

At the time, I was working at IBM so I needed a tool that worked for Java applications, and they didn't look kindly on potentially confidential information being stored outside of the IBM network, so I created my own version of Exception that I named IBMExceptional (you can read a little bit about it and see some screenshots on my IBM portfolio page).  The site for monitoring the errors was a Rails application, but I created a Java client for it that provided 2 reporting options.  

First, I provided an IBMExceptional exception class that could be extended for application specific exceptions.     

        class MyProjectException extends IBMExceptional {
            @SuppressWarnings("unused")
            public MyProjectException() {
                super();
            }

            public MyProjectException(String message, Throwable cause) {
                super(message, cause);
            }

            public MyProjectException(String message) {
                super(message);
            }

            @SuppressWarnings("unused")
            public MyProjectException(Throwable cause) {
                super(cause);
            }
        }

This base exception class would report any instances of thrown exceptions to IBMExceptional.

        try {
            throw new MyProjectException("This is a test exception.");
        } catch (MyProjectException e) {
            // Handle exception as desired, but it was reported to IBM Exceptional
        }

The advantage of this option is that the exception reporting is transparent and automatic.  The disadvantage is that only exceptions that are subclassed from the IBMExceptional base class will be reported. 

Second, I provided a log appender, IBMExceptionalAppender, so that any logged errors would be reported to IBMExceptional.  You added a few configuration lines to log4j.properties like

        log4j.rootCategory=info, exceptional  
        log4j.appender.exceptional=com.ibm.IBMExceptionalAppender

Then calls to the logger with an exception would be reported to IBMExceptional.

        try {
            throw new NullPointerException("Fake Null pointer");
        } catch (NullPointerException e) {
            logger.error("Error description", e);
        }

The advantage of this option is that all exception types can be reported.  The disadvantage is that it requires that the exceptions are logged by the programmer.

This worked well and was all well and good while I remained at IBM, but I have since moved on.  For my current job at USIC, I've led the development of a Java API server using Spring Boot (feeding Angular UIs) and a C# .NET (Framework) application and decided I would similarly like to easily monitor when errors occur on the servers, but this time, instead of having some web site to monitor the errors, I thought it would be much simpler and be sufficient to just send them to dedicated Slack channels.  

For Java we went the route of a log appender, for .NET we could tie into the System.Web.HttpApplication Application_Error handler.

Java Spring Boot application with log appender

Spring Boot uses logback rather than log4j, so in logback-spring.xml we defined a new appender for errors.

	<appender name="SLACK" class="com.usicllc.jsonapi.log.SlackAppender">
		<profile>${profile}</profile>
		<filter class="ch.qos.logback.classic.filter.LevelFilter">
			<level>ERROR</level>
			<onMatch>ACCEPT</onMatch>
			<onMismatch>DENY</onMismatch>
		</filter>
	</appender>

Then the SlackAppender class utilizes the JSlack library and looks like:

public class SlackAppender extends AppenderBase<ILoggingEvent> {
	@Override
	protected void append(ILoggingEvent event) {

		String errorMessage = event.getFormattedMessage();
		StringBuffer message = new StringBuffer("*Error: " + errorMessage + "*\n\n");
		String att = null;

		Map<String, String> mdcMap = event.getMDCPropertyMap();
		if (mdcMap != null) {
			String reqUrl = mdcMap.get("REQUEST_URL");
			String reqMethod = mdcMap.get("REQUEST_METHOD");
			String reqQueryString = mdcMap.get("REQUEST_QUERY_STRING");
			String reqUserAgent = mdcMap.get("REQUEST_USER_AGENT");
			String reqRealIp = mdcMap.get("REQUEST_X_REAL_IP");
			String reqUser = mdcMap.get("REQUEST_USER");
			String reqBody = mdcMap.get("REQUEST_BODY");

			if (reqMethod != null) {
				message.append("*REQUEST_METHOD:* " + reqMethod + "\n");
				message.append("*REQUEST_URL:* " + reqUrl + "\n");
				if (reqQueryString != null) {
					try {
						message.append("*REQUEST_QUERY_STRING:* " + URLDecoder.decode(reqQueryString, "UTF-8") + "\n");
					} catch (UnsupportedEncodingException e) {
					}
				}
				if (reqBody != null) {
					message.append("*REQUEST_BODY:* " + reqBody + "\n");
				}
				message.append("*REQUEST_USER_AGENT:* " + reqUserAgent + "\n");
				message.append("*REQUEST_X_REAL_IP:* " + reqRealIp + "\n");
				message.append("*REQUEST_USER:* " + reqUser + "\n");
			}
		}

		// If we have an exception being reported, then add it to the message
		//
		if (event.getThrowableProxy() != null) {
			StackTraceElementProxy[] stArray = event.getThrowableProxy().getStackTraceElementProxyArray();
			if (stArray.length > 0) {
				// first 10 lines of the stack trace are added to the messages as a code block
				// Full stack trace is also added as an attachment
				message.append("*Stack Trace:*\n```");
				String exceptionMessage = event.getThrowableProxy().getMessage();
				StringBuffer stackTrace = new StringBuffer("*Error Message: " + exceptionMessage + "*\n");

				for (int i = 0; i < stArray.length; i++) {
					StackTraceElement t = stArray[i].getStackTraceElement();
					String traceLine = t.toString();
					if (i < 10) {
						// 1st 10 lines are added to the main message in a code block
						message.append(traceLine + "\n");
					}
					if (traceLine.startsWith("com.usicllc.jsonapi")) {
						// emphasize where the error was in our code
						stackTrace.append("*" + traceLine + "*\n");
					} else {
						stackTrace.append(traceLine + "\n");
					}
				}

				if (stArray.length > 10) {
					message.append("...\n");

					// Full stack trace is more than the 10 lines shown (usually the case)
					// Add the full stacktrace as an attachment
					Attachment attachment = Attachment.builder()
							.title("Full Stack Trace")
							.text(stackTrace.toString())
							.build();
					att = "[{\"title\" :\""+attachment.getTitle()+":\n"+"\",\"text\": \""+attachment.getText().replace("\\", "\\\\")+"\"}]";
				}
				message.append("```");
			}

		}
		String payload = "{\"channel\": \"" + this.getSlackChannel() 
			+ "\",\"username\": \"jsonapi\",\"text\": \""
			+ message.toString().replace("\\", "\\\\") 
			+ "\",\"attachments\":" + att + "}";

		WebhookResponse webhookResponse;
		try {
			webhookResponse = Slack.getInstance().send(
				this.getSlackUrl(), payload);
			if (webhookResponse.getCode() != 200) {
				logger.debug("Unable to send Slack Exception - http error:" + webhookResponse.toString());
			}
		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}

The stack trace is added both with the first 10 lines in a code block in the message body and then the full stack trace as an attachment because its more readable as a code block due to no line wrapping and the use of a monospaced font, but Slack will limit the message size and most stack traces will exceed the limit so adding the full stack trace an attachment provides access to everything albeit in a less readable format.  

Now, it could have just logged the exception message and stack trace, but it's certainly more helpful to know what request caused the problem and that's where we took advantage of the logback MDC (Mapped Diagnoatic Context) to retrieve request data in a Spring GenericFilterBean named PreRequestProcessingFilter Spring will inject this into the request filter chain and the filter code looks like:

@Component
public class PreRequestProcessingFilter extends GenericFilterBean implements CurrentUser {
	Logger logger = LoggerFactory.getLogger(PreRequestProcessingFilter.class);

	@SuppressWarnings("unchecked")
	@Override
	public void doFilter(ServletRequest req, ServletResponse res, FilterChain filterChain) {
		try {
			String bodyJson = null;
			RequestWrapper wrapper = null;
			String method = ((HttpServletRequest) req).getMethod();
			String contentType = ((HttpServletRequest) req).getContentType();

			// Get a copy of the request body for JSON POST and PATCH requests
			if (contentType != null &&
					(contentType.startsWith("application/json") || contentType.startsWith("application/vnd.api+json")) &&
					(method.equalsIgnoreCase("POST") || method.equalsIgnoreCase("PATCH"))) {
				wrapper = new RequestWrapper((HttpServletRequest) req);
				byte[] body = StreamUtils.copyToByteArray(wrapper.getInputStream());
				Map<String, Object> jsonRequest = new ObjectMapper().readValue(body, Map.class);
				bodyJson = jsonRequest.toString();
			}

			String user = getUserName();
			StringBuffer requestURL = ((HttpServletRequest) req).getRequestURL();
			if (requestURL != null) {
				MDC.put("REQUEST_URL", requestURL.toString());
			}

			MDC.put("REQUEST_METHOD", method);
			MDC.put("REQUEST_QUERY_STRING", ((HttpServletRequest) req).getQueryString());
			MDC.put("REQUEST_USER_AGENT", ((HttpServletRequest) req).getHeader("User-Agent"));
			MDC.put("REQUEST_X_REAL_IP", ((HttpServletRequest) req).getHeader("X-Real-IP"));
			MDC.put("REQUEST_USER", user);
			MDC.put("REQUEST_BODY", bodyJson);
			filterChain.doFilter((wrapper != null ? wrapper : req), res);

		} catch (IOException | ServletException e) {
			e.printStackTrace();
		} finally {
			MDC.clear();
		}
	}
}

The big trick in that filter is obtaining the POST or PATCH request body without consuming it and thereby prevening the regular request processing from getting it. Normally, you can only read the request body once. So we create a RequestWrapper and a ServletInputStreamWrapper which allow us to extract a copy of the request's input stream (body). The code for these classes looks like:

public class RequestWrapper extends HttpServletRequestWrapper {

    private byte[] body;

    public RequestWrapper(HttpServletRequest request) throws IOException {
        super(request);

        this.body = StreamUtils.copyToByteArray(request.getInputStream());
    }

    @Override
    public ServletInputStream getInputStream() throws IOException {
        return new ServletInputStreamWrapper(this.body);

    }
}


public class ServletInputStreamWrapper extends ServletInputStream {
    private InputStream inputStream;

    public ServletInputStreamWrapper(byte[] body) {
        this.inputStream = new ByteArrayInputStream(body);
    }

    @Override
    public boolean isFinished() {
        try {
            return inputStream.available() == 0;
        } catch (Exception e) {
            return false;
        }
    }

    @Override
    public boolean isReady() {
        return true;
    }

    @Override
    public void setReadListener(ReadListener listener) {
        // no need to implment this method
    }

    @Override
    public int read() throws IOException {
        return this.inputStream.read();
    }
}

With all this in place, we now get nice Slack messages for errors providing request data and stack trace.  The final result looks like:



.NET Framework Application with Application_Error handler

For the .NET Framework application its much simpler. In our Global.asax.cs file (where we create our application subclass of System.Web.HttpApplication) we define the Application_Error hook which allows us access to the request data, and then we simply send it to Slack utilizing the Slack.Webhooks package.

    public class MvcApplication : System.Web.HttpApplication
    {
        private static readonly bool slackEnabled = Convert.ToBoolean(ConfigurationManager.AppSettings["slackEnabled"]);
        private static readonly string slackURL = ConfigurationManager.AppSettings["slackURL"];
        private static readonly string slackChannel = ConfigurationManager.AppSettings["slackChannel"];
        private static readonly SlackClient slackClient = new SlackClient(slackURL);     
        private static readonly string environment = ConfigurationManager.AppSettings["environmentMode"];

        protected void Application_Error(object sender, EventArgs e)
        {
            Exception ex = null;
            // make sure whatever we do here doesn't generate any exceptions and generate a loop
            try
            {
                ex = Server.GetLastError();
                
                string requestType = HttpContext.Current.Request.RequestType;
                string requestURL = Convert.ToString(HttpContext.Current.Request.Url);
                string requestReferrer = Convert.ToString(HttpContext.Current.Request.UrlReferrer);
                string requestBrowser = HttpContext.Current.Request.Browser.Browser + " " + HttpContext.Current.Request.Browser.Version;
                string requestPlatform = HttpContext.Current.Request.Browser.Platform;
                bool requestIsAuthenticated = HttpContext.Current.Request.IsAuthenticated;
                string requestUserAgent = HttpContext.Current.Request.UserAgent;
                string currentUser = User.Identity.Name?.Split('\\')[1];
                string requestUserAddress = HttpContext.Current.Request.UserHostAddress;
                string requestForwardedForAddress = HttpContext.Current.Request.Headers.Get("X-Forwarded-For");

                Stream requestInputStream = HttpContext.Current.Request.InputStream;
                requestInputStream.Seek(0, SeekOrigin.Begin);
                string requestBody = new StreamReader(requestInputStream).ReadToEnd();

                string logInfoString =
                      "\n   User: " + currentUser
                    + "\n   Request type: " + requestType
                    + "\n   Request URL: " + requestURL
                    + "\n   Referrer: " + requestReferrer
                    + "\n   Browser: " + requestBrowser
                    + "\n   Platform: " + requestPlatform
                    + "\n   UserAgent: " + requestUserAgent
                    + (requestIsAuthenticated ? "" : ("\n   IsAuthenticated: " + requestIsAuthenticated))
                    + "\n   Source address: " + requestUserAddress + (string.IsNullOrEmpty(requestForwardedForAddress) ? "" : 
                      (" / " + requestForwardedForAddress));

                string logInfoString = requestInfoString +
                      (string.IsNullOrEmpty(requestBody) ? "" : ("\n   Request body: " + requestBody));

                log.Error(logInfoString, ex);

                SendSlackMessage("Application Error:", requestInfoString, ex, requestBody);

            }
            catch (Exception exc) {
                // If we have issues getting any of the info above, at least log the exception
                try
                {
                    log.Error("Original exception: ", ex);
                    log.Error("Additionally, there was an exception in Application_Error: " + exc);
                }
                catch (Exception) { }      
            }
        }

        public static void SendSlackMessage(string title, string message, Exception exception, string requestBody)
        {
            // don't send to slack if disabled or running locally
            if (!slackEnabled || environment == "local") return;

            // add time and make the title bold so that messages posted in quick succession can be easily told apart
            // (*stuff* makes it bold)
            string slackText = "*" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + (title != null ? (" - " + title) : "") + "*";

            if (message != null)
            {
                slackText += "\n" + message;
            }

            List<SlackAttachment> slackAttachments = new List<SlackAttachment>();

            if (!string.IsNullOrWhiteSpace(requestBody))
            {
                var requestBodyAttachment = new SlackAttachment
                {
                    Text = requestBody
                };

                slackAttachments.Add(requestBodyAttachment);
            }

            if (exception != null)
            {
                var slackAttachment = new SlackAttachment
                {
                    Text = exception.ToString(),
                    Color = "#C20202"
                };

                slackAttachments = new List<SlackAttachment> { slackAttachment };
            }

            // local defaults. Comment out first line of the method to test locally if needed
            string slackUser = ".NET Application";
            string slackUserEmoji = Emoji.Wrench;

            var slackMessage = new SlackMessage
            {
                Channel = slackChannel,
                Text = slackText,
                IconEmoji = slackUserEmoji,
                Username = slackUser,
                Attachments = slackAttachments
            };

            slackClient.PostAsync(slackMessage);
        }



Monday, June 28, 2021

The CVS ExtraCare Coupon Clipper

In the same spirit as the Publix Digital Coupon Clipper for which I've received a lot of positive feedback and requests for similar functions for other stores, I also announce the CVS ExtraCare coupon clipper. These digital coupon sites don't like to provide a "Clip All" function.  I presume this is for some advertising purpose so you're forced to look at each one.  I like to have all the coupons clipped and if I happen to buy something there's a coupon it will just apply it at checkout. 

Plus with CVS being notorious for spitting out a whole roll of receipt paper with printed coupons, if they're all pre-clipped electronically on your CVS ExtraCare card I don't believe it will print them so you can save some trees. 

Please go to https://pothoven.net/CvsClipper.html for installation instructions, and please drop me a comment here if you try it out to let me know how it works for you.

Return of the Publix Coupon Clipper

My Publix Coupon Clipper bookmarklet has been offline for a while now due to changes that had been made to the to website code to disable it.  I like to think the changes made were specifically targeted at my bookmarklet, but it may have been purely coincidental too.  Regardless, my handy clipper was rendered inoperable with no way to circumvent what they had done to disable it.

For those reading this who are unfamiliar with it, I introduced the coupon clipper in January 2014 as a "Clip All" function that was (and still is) missing from the site to quickly and easily clip all the digital coupons so they'll all be ready and waiting when you checkout.  I revised it November 2015 to deal with some page changes, but in December of 2016 a change was added to the site that permanently disabled it.

Fortunately for us, they've re-written the coupon site using Vue instead of JQuery and the code that prevented the coupon clipper from working did not make the transition.  I've made the necessary changes to make it work with the new page, and it's working again!  So while it lasts...

Please go to https://pothoven.net/PublixClipper.html for installation instructions, and please drop me a comment here if you try it out to let me know how it works for you.

Wednesday, December 09, 2020

Angular Clipboard Service

It's about time I post something new! Since my last post I've had to learn and build applications in the Ember, React, and now Angular JavaScript Frameworks.  Along the way I've learned some interesting tips and tricks I can share here.


I needed the ability to place content in the clipboard, but needed it lots of placed throughout the application. So here's a simple Clipboard service that will place any string into the clipboard.  I incorporated the use of MatSnackBar to notify the user that the value has been placed in the clipboard.

import {Injectable, Renderer2, RendererFactory2} from '@angular/core';
import { MatSnackBar } from '@angular/material/snack-bar';

@Injectable({
  providedIn: 'root'
})
export class ClipboardService {
  private renderer: Renderer2;

  constructor (
      private rendererFactory: RendererFactory2,
      private snackBar: MatSnackBar,
      ) {
    // Get an instance of Angular's Renderer2
    this.renderer = this.rendererFactory.createRenderer(null, null);
  }

  copy(value: string) {

    // create an temporary text input field to contain the value and select the value
    const input = this.renderer.createElement('input');
    this.renderer.setProperty(input, 'value', value);
    this.renderer.appendChild(document.body, input);
    input.focus();
    input.select();
    input.setSelectionRange(0, 99999); // For mobile devices

    // Copy the selected text inside the text field to the clipboard
    document.execCommand('copy');
    this.snackBar.open(`Copied "${value}" to clipboard`, undefined, { duration: 1000 });

    // remove the temporary text input field
    this.renderer.removeChild(document.body, input);
  }
}

You then simply inject it into any component that needs it:

import { ClipboardService } from 'clipboard.service';

constructor( private clipboard: ClipboardService ) {} 

and then copy whatever values you want to the clipboard simply as:
this.clipboard.copy(value);

Monday, May 13, 2019

2's complement and BCD hex values in JavaScript

I'm sure this is will not be a highly accessed post as these are not a commonly required functions, but in my current job, we process data from IoT GPS devices which send data in compact formats.  To process this data using Node.js in an AWS Lambda function, I had the need to convert a 2's complement hex values and BCD (binary-coded decimal) hex value to regular numbers in JavaScript.  The examples I found were in Java, so I converted it to JavaScript and am sharing here in case anyone else needs it.

       const negativeHexTestPattern = /^[89ABCDEF]/i


        /* parse signed 2's complement hex string to number */
        hexToTwosComplement(hex) {
                let result = parseInt(hex, 16)

                // Check if high bit is set.
                if (negativeHexTestPattern.test(hex)) {
                        // Negative number
                        const subtrahend = (2 ** (hex.length * 4))
                        result -= subtrahend
                }
                return result
        }


        /* parse packed binary-coded decimal (BCD) format where unused
         * trailing digits (4-bits) are filled with all ones (1111). */
        hexBCDToString(hex) {
                let decoded = ''
                for (let i = 0; i < hex.length; i++) {
                        if ((parseInt(hex[i],16) & 0x0F) !== 0x0F) {
                                decoded += hex[i]
                        } else {
                                break
                        }
                }
                return decoded
        }

Here's a few test cases to demonstrate.
                it('converts positive hex values to twos complement', () => {
                        const val = new Processor().hexToTwosComplement('13f6b4eb')
                        chai.expect(val).to.equal(334935275)
                })

                it('converts negative hex values to twos complement', () => {
                        const val = new Processor().hexToTwosComplement('ec094b15')
                        chai.expect(val).to.equal(-334935275)
                })

                it('converts negative 2 byte hex values to twos complement', () => {
                        const val = new Processor().hexToTwosComplement('ffae')
                        chai.expect(val).to.equal(-82)
                })

Wednesday, March 27, 2019

Templated version of JavaScript Object.assign

In a recent article I provided a revised version of the JavaScript Object.assign function that also merged nested objects.  In this article, I'm going to revise it slightly so that it only assigns attributes according to a template.  The deepAssign method is useful for merging in partial data, some of which may be nested, into a larger set of (state) data.   The templateAssign is useful for pruning the data you're going to store to only what you need / care about.

Why would you need to do this?


In GraphQL you can specify the attributes you want so probably don't need to, but in other techniques (like json:api) you get the full object which may contain way more data than you need, and if the data set it large it can consume a lot of memory unnecessarily.  In the application this function is derived from, we have a lot of data and I've seen quite a few "Out of memory" errors in our application monitoring.  So eliminating unnecessary data is valuable.

The standard Object.assign (and the Object.deepAssign) method will combine all the object attributes, but if the incoming data contains extra attributes you don't care about, it can be handy to prune it to only what you care about.  This templateAssign function will accept a template object as the first argument and only copy over attributes from the additional source objects that are defined in the template.

If this is to be used in a map-reduction system like Redux, be sure to clone the template object into a new object via Object.assign.

Example:
Let's assume we only want attribute "a" either at the top level or a nested level

> const template =  { a: undefined, deep: { a: undefined } }
> const obj1 = { a : "a", deep: { a: "a" } }
> const obj2 = { b : "b", deep: { a: "b", b: "b" } }
> const obj3 = { c : "c", deep: { b: "b", c: "c" } }

Standard assign, doesn't merge nested "deep" object, and keeps all (top-level) attributes
> Object.assign({}, obj1, obj2, obj3)
{ a: 'a', deep: { b: 'b', c: 'c' }, b: 'b', c: 'c' }

deepAssign, merges everything (top-level and nested objects)
> Object.deepAssign({}, obj1, obj2, obj3)
{ a: 'a', deep: { a: 'b', b: 'b', c: 'c' }, b: 'b', c: 'c' }

templateAssign, merges all levels, but only keeps attributes in the template
> Object.templateAssign(Object.assign({}, template), obj1, obj2, obj3)
{ a: 'a', deep: { a: 'b' } }

Here's the code:

if (!Object.prototype.templateAssign) {
    Object.prototype.templateAssign = function(...objs) {
        let target = objs.shift();
        let source = objs.shift();
        
        if (source) {
            for(const attribute in source) {
                if (attribute in source &&  typeof(source[attribute]) === "object") {
                    target[attribute] = Object.templateAssign(Object.assign({}, target[attribute] || {}), source[attribute]);
                } else if (attribute in target &&
                           source.hasOwnProperty(attribute) &&
                           source[attribute] !== undefined) {
                    target[attribute] = source[attribute];
                }
            }
        }
        if (objs.length > 0) {
            return Object.templateAssign(target, ...objs);
        } else {
            return target;
        }
    };
}

Tuesday, March 26, 2019

Make NVM work like RVM

I've used RVM (Ruby Version Manager) for years and it has a great feature of automatically switching your Ruby version as you navigate to to project folders to use the Ruby version specified for that project. For NVM (Node Version Manager) you have to manually tell nvm to switch node versions via nvm use.    If you add the following code to your bash configuration, nvm will switch automatically like rvm does when it finds a .nvmrc file.  (note: I am not the original author of this code.  I tweaked it from another source, but I don't recall where that was).

# fix NVM to work like RVM
#
find-up () {
    path=$(pwd)
    while [[ "$path" != "" && ! -e "$path/$1" ]]; do
        path=${path%/*}
    done
    echo "$path"
}

cdnvm(){
    cd $@;
    nvm_path=$(find-up .nvmrc | tr -d '[:space:]')
 
    # If there are no .nvmrc file, use the default nvm version
    if [[ ! $nvm_path = *[^[:space:]]* ]]; then
  
        declare default_version;
        default_version=$(nvm version default);
  
        # If there is no default version, set it to `node`
        # This will use the latest version on your machine
        if [[ $default_version == "N/A" ]]; then
            nvm alias default node;
            default_version=$(nvm version default);
        fi
  
        # If the current version is not the default version, set it to use the default version
        if [[ $(nvm current) != "$default_version" ]]; then
            nvm use default;
        fi
  
    elif [[ -s $nvm_path/.nvmrc && -r $nvm_path/.nvmrc ]]; then
        declare nvm_version
        nvm_version=$(<"$nvm_path"/.nvmrc)
  
        # Add the `v` suffix if it does not exists in the .nvmrc file
        if [[ $nvm_version != v* ]]; then
            nvm_version="v""$nvm_version"
        fi
  
        # If it is not already installed, install it
        if [[ $(nvm ls "$nvm_version" | tr -d '[:space:]') == "N/A" ]]; then
            nvm install "$nvm_version";
        fi
  
        if [[ $(nvm current) != "$nvm_version" ]]; then
            nvm use "$nvm_version";
        fi
    fi
}
alias cd='cdnvm'

Monday, March 25, 2019

Activity LEDs

In some of my applications I get ongoing data from websockets and I want some visual indication that the websocket is connected and receiving data, so I added an LED indicator much like a network router or switch would have.

Whereever I want the LED, I add div to show the LED.

<div class="led" id="ws-led">
<div class="led-red-off">
</div>
</div>

then add some CSS rules to make it look like an LED indicator

div.led{
 display     : inline-block;
 vertical-align    : bottom;
}

div.led-red-off,
div.led-red-on,
div.led-green-off,
div.led-green-on {
 border     : 0;
 border-radius    : 50%;
 height     : 1em;
 width     : 1em;
 vertical-align    : middle;
 background-repeat   : no-repeat;
 display     : inline-block;
}

div.led-red-on {
 background    : #F44336; opacity: 1;
}
div.led-green-on {
 background    : #4CAF50; opacity: 1;
}
div.led-red-off {
 background    : #F44336; opacity: .5;
}
div.led-green-off {
 background    : #4CAF50; opacity: .5;
}

and finally, some JS code to make them blink.

    var wsLED = document.getElementById('ws-led');

    // connected to the Websocket server (general)
    function connected(greeting) {
        // change the LED from initial red to green
        if (wsLED) wsLED.firstChild.setAttribute("class", "led-green-off");
    }

    // connection to Websocket server lost
    function disconnected() {
        // if we loose the websocket connection, change the LED to red
        if (wsLED) wsLED.firstChild.setAttribute("class", "led-red-on");
    }

    /**
     * logWSActivity
     *
     * log some WS activity
     */
    function logWSActivity(...args) {
        console.log(...args);
        flashLED(wsLED);
    }

    /**
     * flashLEDs
     *
     * The visual effect of the flashing LEDs is done by switching the
     * CSS class for a 10th of a second per activity.  If a new
     * activity happens within that time, the timeout to turn it back
     * off is reset to a new 10th of a second.
     */
    var ledOffTimeout;
    function flashLED(led) {
        // flash activity LED
        if (led) {
            if (ledOffTimeout) { clearTimeout(ledOffTimeout); }
            led.firstChild.setAttribute("class", "led-green-on");
            ledOffTimeout = setTimeout(function() {
                led.firstChild.setAttribute("class", "led-green-off"); }, 100);
        }
    }

JavaScript String functions (toCamelCase, dasherize, and titleize)

I had the need for a few String functions in JavaScript to manage some variances between data sources so I figured I'd share them.  For example, from one source attributes in the JSON data would use underscores to separate words and in another it would uses dashes (my_attribute vs my-attribute).  Also for simple data display, I wanted to titleize the attribute name for a label (My Attribute). 


/**
 * String.toCamelCase
 *
 * Convert a string to camel case, including hyphens and underscores
 */
String.prototype.toCamelCase = function() {
    return this.replace(/^([A-Z])|[\s-_](\w)/g, function(match, p1, p2, offset) {
        if (p2) return p2.toUpperCase();
        return p1.toLowerCase();        
    });
};

/**
 * String.dasherize
 *
 * Dasherize a string, including periods and underscores
 */
if (!String.prototype.dasherize) {
    String.prototype.dasherize = function() {
        return this.replace(/^([A-Z])|[\s\._](\w)/g, function(match, p1, p2, offset) {
            if (p2) return "-" + p2.toLowerCase();
            return p1.toLowerCase();        
        });
    };
}

/**
 * String.capitalize
 *
 * Capitalize the first letter of a string
 */
if (!String.prototype.capitalize) {
    String.prototype.capitalize = function() {
        return this.charAt(0).toUpperCase() + this.slice(1);
    };
}

/**
 * String.titleize
 *
 * Capitalize the first letter of every word in a string, also separates
 * words by spaces if formally separated by dashes or underscores
 *
 */
if (!String.prototype.titleize) {
    String.prototype.titleize = function() {
        return this.replace(/^([A-Z])|[\s-_](\w)/g, function(match, p1, p2, offset) {
            if (p2) return " " + p2.toUpperCase();
            return p1.toLowerCase();        
        }).capitalize();
    };
}

Deep Copy version of Javascript Object.assign

I was working on some Redux work and needed a reducer that would merge in some sparse updates to the current state of an object.

If you're learning Redux you may be familiar with the tutorial example of a TODO list item where it's changing one attribute of the TODO list:

return Object.assign({}, state, {visibilityFilter: action.filter});

Now, instead of changing a single top-level attribute like the visibility filter, assume you have some data that needs to be merged into the existing object both at the top level and in some nested attributes.  For example, presume your current state looks like:

{ a : "a", deep: { a: "a" }

and you get new data for that state that need to be merged in that looks like:

{ b : "b", deep: { b: "b" } }

The Object.assign function will do a shallow copy merge and the result will be:

> Object.assign({}, { a : "a", deep: { a: "a" } }, { b : "b", deep: { b: "b" } })
{ a: 'a', deep: { b: 'b' }, b: 'b' }

The new value of deep simply replaced the first value.  The nested value of deep wasn't merged together.  The method below will correctly merge nested values as follows:

> deepAssign({}, { a : "a", deep: { a: "a" } }, { b : "b", deep: { b: "b" } })
{ a: 'a', deep: { a: 'a', b: 'b' }, b: 'b' }

Here's the code:

function deepAssign(...objs) {
  const target = objs.shift();
  const source = objs.shift();

  if (source) {
    if (source instanceof Array) {
      for (const element of source) {
        if (element instanceof Array) {
          target.push(deepAssign([], element));
        } else if (element instanceof Object) {
          target.push(deepAssign({}, element));
        } else {
          target.push(element);
        }
      }
    } else {
      for (const attribute in source) {
        // eslint-disable-next-line no-prototype-builtins
        if (source.hasOwnProperty(attribute) && source[attribute] !== undefined) {
          if (source[attribute] instanceof Array) {
            target[attribute] = target[attribute] || [];
            for (const element of source[attribute]) {
              if (element instanceof Array) {
                target[attribute].push(deepAssign([], element));
              } else if (element instanceof Object) {
                target[attribute].push(deepAssign({}, element));
              } else {
                target[attribute].push(element);
              }
            }
          } else if (source[attribute] instanceof Object) {
            if (source[attribute].toString() === '[object Object]') {
              // simple data object so deep copy it
              target[attribute] = deepAssign((typeof target[attribute] === 'object') ? target[attribute] : {}, source[attribute]);
            } else {
              // instance of some class, so just copy over the object
              target[attribute] = source[attribute];
            }
          } else {
            target[attribute] = source[attribute];
          }
        }
      }
    }
  }

  if (objs.length > 0) {
    // eslint-disable-next-line @typescript-eslint/no-unused-vars
    return deepAssign(target, ...objs);
  } else {
    return target;
  }
};


Update 7/15/19 : Original version didn't handle Arrays correctly.  They become objects.  They now copy correctly including nested array and object in the arrays.  You can't just use the spread operator (newArray = [...oldArray]) to copy it or it wouldn't copy the nested objects as new objects

Update 8/27/21 : Objects that weren't simple data objects were being lost, so now complex objects are brought over to the new copy.

Tuesday, April 05, 2016

Re-enable disabled disable_with fields

I have some forms that do JavaScript based validation before submitting to the server.  The submit button is using the disable_with construct, so when the submit button is pressed, it is correctly disabled, but when the client-side validation fails, it's stuck disabled.

To remedy this situation, I added these lines to the JavaScript code that hides the error messages after they've been displayed for 5 seconds.

// re-enable 'disable_with' fields after validation errors
var disabled_with = jQuery('[data-disable-with][disabled]');
if (disabled_with) {
    jQuery.rails.enableElement(disabled_with);
    disabled_with.val(function(){ return jQuery(this).text()}).prop('disabled', false);
}

The first line is the commonly proposed solution that I found elsewhere online, but when I tried it (in Chrome), I found that it didn't work, leaving the button disabled with the disabled text displayed. However, one important thing it did was to insert the original button text as HTML content of the input element.  I added the second line to use that text as the button value and then actually enable the button.


Saturday, January 16, 2016

Setup of Ruby Development Environment VM

I recently needed to provide some development environment setup instructions to a friend, so I figured it would be a good idea to document them here in case I need them again or if it helps anyone else.

Let me start off with a few premises:

  • I like doing Ruby on Rails development.
  • Ruby development on Windows is not pretty.
  • The computer I'm using most often is Windows-based (provided for my "real" job which is not Ruby development).
  • My personal computers are Macs.
  • I want to be able to work from any computer but don't want to hassle with keeping the environments in sync.
  • I've used multiple flavors of Linux over the years back to the SLS days (Linux version 0.9x), and am very comfortable with it.
With those premises in mind, I've adopted the use of a virtual machine (VM) to be my development environment.  It allows me to separate my personal development environment from other clutter (or even probing by workplace compliance scans), and allows me to readily move it from machine to machine as necessary.  I also usually host my applications on Linux-based servers, so it provides me more confidence in an environment that matches the deployment environment.

I personally don't like living in a VM window to work.  I also feel that I'm already running a windows manager (Windows or OSX) on my host operating system, so why incur the overhead of running another one in the VM.  Therefore, I run a "headless" Linux in the VM, and open X windows on the host by running a simple X server on the host OS.

Step 1 - Setup an X server on the host OS

For Windows, I recommend VcXsrv (see http://sourceforge.net/projects/vcxsrv/), but if you're already using Cygwin for other UNIX applications, the Cygwin Xserver (see http://x.cygwin.com/docs/ug/setup-cygwin-x-installing.html) works well, or alternatively you could try Xming (http://sourceforge.net/projects/xming/files/latest/download).

For OSX, I use XQuartz (see http://www.xquartz.org/)

Step 2 - Install VMWare (or VirtualBox, Parallels, etc.)

I'm not going to provide instructions for every virtualization software out there.  I use VMWare so that's what I'm describing.  Other solutions work just as well.

For Windows, VMWare Player does everything you need, and it's free!

For OSX, there's no free player, so you have to purchase VMWare Fusion.  (but, hey, it's the only thing you need to buy!).

Step 3 - Download a Linux distribution ISO

Again, there are plenty of options (Ubuntu, Red Hat, SuSE, etc), but I'm using Ubuntu here as it is frequently used for hosting.  You want the Server version (not Desktop) as it will be more light-weight without all the X11 + window manager overhead.  Also, I recommend the LTS (longer term service) version so you don't have to deal with upgrading your OS all the time (besides security fixes of course).

Step 4 - Create a New Virtual Machine

Select the VMWare option to create a new virtual machine, point it to the ISO and it should be pretty straight forward.

You don't have to be excessive with the resources you allocate for the VM.  The default recommendations by VMWare will probably suffice, but if you have a little more to give, then bump it up a little.  For example, the VMWare recommendation for the memory is 1GB, my system has 16GB so I gave the VM 2GB.  I allowed it 4 out of the 8 processor cores, and give it up to 40GB of disk space.

Step 5 - Forward local ports to VM

While this is not strictly necessary, it is useful.

For Windows, in earlier versions of VMWare Player, the Virtual Network Editor was included, but you needed to know the right command to access it.  In case the current omission is unintentional and they add the network editor back in future releases, the method to access the network editor for VMPlayer was:

  1. Open command prompt as administrator
  2. cd "C:\Program Files (x86)\VMware\VMware Player"
  3. rundll32.exe vmnetui.dll VMNetUI_ShowStandalone

As it stands currently, that will give you an error that the vmnetui.dll is not found.  So instead, you need to extract the Virtual Network Editor from the VMWare Workstation package.  This is done by:

  1. Download the latest VMWare Workstation package.
  2. unpack it locally with this command :
    "
    VMWare-worksation-full-10.0.1-xxxxxx.exe /e .\ext"
    note the xxxxxx is the release id you got at download
  3. Go to the newly created "ext" directory and open the "core.cab" (use your favorite zip util)
  4. Get the "vmnetcfg.exe" from there, and copy it to the vmwareplayer install directoryGet the "_vmnetcfglib.dll" file and rename it "vmnetcfglib.dll", and then copy it to the same directory than for vmnetcfg.exe
OR  

Download vmnetcfg.exe and vmnetcfglib.dll and put them in your VMWare Player install directory.

Once you have the Virtual Network Editor, select the NAT adapter (should be VMnet8), click on the "NAT Settings..." button, and add whatever ports you'll be using as shown below.  In this example, my VM IP is 192.168.216.131 and I'm forwarding ports 22 for SSH and 3000 as the default Rails server port.  Add port 80 if you'll be running your apps through a normal HTTP server like Apache or nginx (ex. via Passenger).




For Macs using VMWare Fusion, edit the vmnet8 NAT config file using the terminal and your favorite editor

sudo emacs /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf

or
 
sudo vim /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf
 
 
Near the bottom you’ll see something like this
 
[incomingtcp]
 
# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
# = :
#8080 = 172.16.3.128:80
 
This is where we’ll be putting the port forwarding.  We'll assuming the same VM IP and that we still want to forward to the Rails server port 3000 and SSH port 22.   Since OSX is likely already running an SSH server on port 22, we'll forward from port 2222 instead.  So, add the lines:

3000 = 192.168.216.131:3000
2222 = 192.168.216.131:22 


Restart VMWare Fusion and it should be forwarding to these ports for you.  Alternatively, you could restart the VMWare networking using the command line:


sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --stop
sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --start


Optionally, if you want to ensure your VM always gets the same IP address, you can edit /Library/Preferences/VMware\ Fusion/vmnet8/dhcpd.conf and add a clause to the bottom with the format of:


host <some-name> {
    hardware ethernet <MAC-ADDRESS>;
    fixed-address <IP-address>;
}

You can determine the VMWare assigned MAC address for your VM by looking in your VM's *.vmx file and look for a line like:

ethernet0.generatedAddress = "00:0c:29:42:f0:54"

Then to assign a static IP address of 192.168.216.131 to this VM, you could add the clause:

host ubuntu {
    hardware ethernet 00:0c:29:42:f0:54;
    fixed-address
192.168.216.131;
}


Remember to restart the Fusion Networking services either by restarting Fusion or using the command line option.
Most of the OSX instructions came from http://encyclopediaofdaniel.com/blog/fusion-dhcp-port-forwarding/ where you can also find information on a Ruby gem to make the changes for you.

Step 6 - Connect using SSH with X forwarding

X11 forwarding needs to be enabled on both the client side and the server side.

On the client side, the -X (capital X) option to ssh enables X11 forwarding, and you can make this the default (for all connections or for a specific conection) with ForwardX11 yes in ~/.ssh/config.

On the server side, X11Forwarding yes must specified in /etc/ssh/sshd_config. Note that the default is no forwarding (some distributions turn it on in their default /etc/ssh/sshd_config), and that the user cannot override this setting.

The xauth program must be installed on the server side. If there are any X11 programs there, it's very likely that xauth will be there. In the unlikely case xauth was installed in a nonstandard location, it can be called through ~/.ssh/rc (on the server!).

Note that you do not need to set any environment variables on the server. DISPLAY and XAUTHORITY will automatically be set to their proper values. If you run ssh and DISPLAY is not set, it means ssh is not forwarding the X11 connection.

To confirm that ssh is forwarding X11, check for a line containing Requesting X11 forwarding in the ssh -v -X output. Note that the server won't reply either way.

(credit to http://unix.stackexchange.com/questions/12755/how-to-forward-x-over-ssh-from-ubuntu-machine)

Friday, November 20, 2015

Updated Clip All function for Publix Digital Coupons

Over a year ago, I provided the Clip All function for Publix Digital Coupons.  It's been working well and I've received plenty of positive feedback for it.

This month, Publix revised their digital coupon site and the function stopped working.  I have fixed my code and it should work again!  There are a few things to point out though.

First, the method I used in the past for attaching my JavaScript code to their site no longer works.  When I do that, it causes the page to re-load itself, which then negates the addition of my code.  I now load the code and invoke the function from within the bookmarklet.  That works out fine, but that method no longer allows me to re-direct you to the coupon site if you're not currently there.  So you need to invoke the code from the coupon site.

Second, the no longer have links to more pages of coupons, but load more dynamically as you scroll down the page.  I try to make the code emulate the scrolling activity to automatically load more coupons, but it doesn't always trigger the event to load more coupons.  So, if you're sitting on the bottom of the page with all the coupons 'clipped', try to scroll up and down a little to see if more coupons start loading.  If more load, the auto-clipping function should kick back into action.

Please go to https://pothoven.net/PublixClipper.html for installation instructions, and please drop me a comment here if you try it out to let me know how it works for you.

Tuesday, January 07, 2014

Bishop Swap Puzzle Fixed!

I implemented a Bishop Swap Puzzle back in 2007 (see my prior post), but it has had a bug in it where bishops of the same color could pass through each other.  I had noted it as a 'TODO' in my code comments while I was writing it, but I had neglected to go back and fix it and forgot all about it until I was trying it out again recently.

I'm happy to say that I've now fixed the bug and it works correctly, so happy gaming!


Once you've completed the Bishop Swap puzzle, be sure to check out the others!

Thursday, January 02, 2014

Clip All function for Publix Digital Coupons

Happy New Year!

Sadly, it's been over a year since my last post.  To make up for not contributing for so long, I'll provide a little gift for those of you who have resolved to save more in the new year.  In particular, you need to live in the southeastern United States and do your grocery shopping at Publix.

Last year, Publix introduced a digital coupon site. This site provides a collection of coupons that you can "clip" and then when you're at the checkout lane, you enter your phone number on the credit card reader keypad and it will automatically apply any coupons you have clipped that apply to the groceries you have purchased.  The problem with the site is that they have not provided a "Clip All" function, so you need to look through several pages of coupons and clip any coupon you're interested in one at a time.  This can be a very time consuming process as many pages of coupons can be added each week.

To rectify this problem, I created a bookmarklet that provides a "Clip All" function.  It will clip all on the current page of coupons and continue to navigate through all available pages of coupons and clip them all.  The bookmarklet also works from mobile devices (at least iPads) so you can even quickly clip all the digital coupons while you're in Publix using their WiFi.

Please go to https://pothoven.net/PublixClipper.html for installation instructions, and please drop me a comment here if you try it out to let me know how it works for you.


Monday, October 29, 2012

Ruby 1.8.7 vs 1.9.3 performance

There are plenty of Ruby 1.8 to 1.9 benchmark results out there, and this is by no means as thorough as most.  I thought I'd share the results for one of my websites when I upgraded it from Ruby 1.8.7 to Ruby 1.9.3 as demonstrated in my New Relic report for the site. Conveniently, I upgraded the site on a Monday morning, and since that's also when New Relic reports switch over, it provided a fairly clean week-to-week comparison (perhaps 1/3 of Monday 10/22 was still Ruby 1.8.7).

High Level Summary

With no other changes, other than fixes required to allow the application to work with Ruby 1.9.3, simply upgrading Ruby resulted in the overall response time average for the week dropped from 304ms to 72ms (a 76.32% drop in response time).

Unfortunately, the week immediately before the upgrade was a bit of an anomaly.  The week before that had a 153ms average response time, which is more in line with typical weekly results.  However, that's still a 53% reduction in response time from 1.8.7 to 1.9.3.

Here's a daily comparison for the 3 weeks:

Ruby 1.8.7 


Ruby 1.9.3



Keep up the good work, Ruby development team!

attachment_fu as a gem for Rails 3.2

As I've mentioned before, I develop the web site for St. Francis Society Animal Rescue.  As I've also described in that earlier article, it started as a Rails 1.2 app, then Rails 2.1, then 2.3.  Right now it's in Rails 3.1, but I'm about to switch it to Rails 3.2.  Having some history, it was developed to use attachment_fu to upload all the images for the cats and dogs.  While I know many people have abandoned attachment_fu for paperclip, or carrierwave, or dragonfly, etc. attachment_fu has continued to work just fine for me so if it ain't broke, don't fix it.  Plus, there are currently over 13,000 animal images that have been uploaded, and I don't really want to hassle with converting them over to a new attachment system.

Enter Ruby 1.9.x

The last official update to the attachment_fu github repository was on April 25, 2009.  While that update is for Ruby 1.9 compatibility fixes, if you try to use it as is for Ruby 1.9.3, you'll find it won't work.  To that end, I've forked off a new repository that will fix additional Ruby 1.9.3 incompatibilities.  If you want to continue to use attachment_fu as a plugin with Ruby 1.9.x and Rails prior to 3.2, you can use my github repository in your project simply by issuing this command in your project:

git submodule add https://github.com/pothoven/attachment_fu.git vendor/plugins/attachment_fu

Enter Rails 3.2.x

attachment_fu has always functioned as a plugin (vendor/plugins), the problem is that Rails 3.2 will give you this error if you continue to use it as a plugin:

DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. 
Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. 
See the release notes for more on this: http://weblog.rubyonrails.org/2012/1/4/rails-3-2-0-rc2-has-been-released.

To that end, I've also updated my fork of attachment_fu to function as a gem! Simply add this line to your Gemfile
 
gem 'pothoven-attachment_fu'

I need to acknowledge Christophe Porteneuve for doing most of the gem conversion work. I just pulled in his updates and fixed a few problems with it that I encountered. Please feel free to let me know if you have any problems using the gem.