Tuesday, August 26, 2014

Existing Configuration Solutions

Existing Application Configuration Solutions

Within this blog I try to give some details on existing common configuration mechanisms in place. If I miss something, let me know, so I can add it here. Also not that the ordering of the section in this blog is completely random, it does not reflect any valuation from my side.

Also I explicitly do not focus on deployment configuration such as EJB deployment descriptors, CDI beans configuration, resource configuration, configuration for JPA, Bean Validation, web applications etc. This may be covered in another blog.

1. Play Framework

See http://www.playframework.com/documentation/2.0/Configuration

The default configuration file of a Play 2.0 application must be defined inconf/application.conf. It uses the HOCON format ( “Human-Optimized Config Object Notation”).

These system properties specify a replacement for application.conf. In the replacement config file, you can use include “application” to include the original default config file; after the include statement you could go on to override certain settings.

HOCON Syntax

Its basically a simplified JSON-Format with some changes, but not compatible with pure JSON.
  • Comments, with # or //
  • Allow omitting the {} around a root object
  • Allow = as a synonym for :
  • Allow omitting the = or : before a { so foo { a : 42 }
  • Allow omitting commas as long as there's a newline
  • Allow trailing commas after last element in objects and arrays
  • Allow unquoted strings for keys and values
  • Unquoted keys can use dot-notation for nested objects, foo.bar=42 means foo { bar : 42 }
  • Duplicate keys are allowed; later values override earlier, except for object-valued keys where the two objects are merged recursively
  • include feature merges root object in another file into current object, so foo { include "bar.json" }merges keys in bar.json into the object foo
  • include with no file extension includes any of .conf.json.properties
  • you can include files, URLs, or classpath resources; use include url("http://example.com") or file()or classpath() syntax to force the type, or use just include "whatever" to have the library do what you probably mean (Note: url()/file()/classpath() syntax is not supported in Play/Akka 2.0, only in later releases.)
  • substitutions foo : ${a.b} sets key foo to the same value as the b field in the a object
  • substitutions concatenate into unquoted strings, foo : the quick ${colors.fox} jumped
  • substitutions fall back to environment variables if they don't resolve in the config itself, so ${HOME} would work as you expect. Also, most configs have system properties merged in so you could use ${user.home}.
  • substitutions normally cause an error if unresolved, but there is a syntax ${?a.b} to permit them to be missing.
  • += syntax to append elements to arrays, path += "/bin"
  • multi-line strings with triple quotes as in Python or Scala
Examples:
foo {
    bar = 10
    baz = 12
}
foo.bar=10
foo.baz=12

2. Typesafe Config Library

  • Formats supported: Java properties, JSON, and a human-friendly JSON superset.
  • merges multiple files across all formats
  • can load from files, URLs, or classpath
  • users can override the config with Java system properties
  • converts types, so if you ask for a boolean and the value is the string "yes", or you ask for a float and the value is an int, it will figure it out
  • Supports substitutions
  • API based on immutable Config instances, for thread safety and easy reasoning about config transformations
  • API Example
Config conf = ConfigFactory.load();
int bar1 = conf.getInt("foo.bar");
Config foo = conf.getConfig("foo");
int bar2 = foo.getInt("bar");
The convenience method ConfigFactory.load() loads the following (first-listed are higher priority):
  • system properties
  • application.conf (all resources on classpath with this name)
  • application.json (all resources on classpath with this name)
  • application.properties (all resources on classpath with this name)
  • reference.conf (all resources on classpath with this name)
The idea is that libraries and frameworks should ship with a reference.conf in their jar. Applications should provide an application.conf, or if they want to create multiple configurations in a single JVM, they could useConfigFactory.load("myapp") to load their own myapp.conf. (Applications can provide a reference.conf also if they want, but you may not find it necessary to separate it from application.conf.)
Finally it supports configuration merging based on so called fallbacks:
Config devConfig = originalConfig
                     .getConfig("dev")
                     .withFallback(originalConfig)

3. Spring


Spring Boot allows you to externalize your configuration so you can work with the same application code in different environments. You can use properties files, YAML files, environment variables and command-line arguments to externalize configuration. Property values can be injected directly into your beans using the @Value annotation, accessed via Spring’s Environment abstraction or bound to structured objects.
Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values, properties are considered in the the following order:
  1. Command line arguments.
  2. Java System properties (System.getProperties()).
  3. OS environment variables.
  4. RandomValuePropertySource that only has properties in random.*.
  5. @PropertySource annotations on your @Configuration classes.
  6. Application properties outside of your packaged jar (application.properties including YAML and profile variants).
  7. Application properties packaged inside your jar (application.properties including YAML and profile variants).
  8. Default properties (specified using SpringApplication.setDefaultProperties).
SpringApplication will load properties from application.properties files in the following locations and add them to the Spring Environment:
  1. /config subdir of the current directory.
  2. The current directory
  3. A classpath /config package
  4. The classpath root
In addition to application.properties files, profile specific properties can also be defined using the naming convention application-{profile}.properties.

The values in application.properties are filtered through the existing Environment when they are used so you can refer back to previously defined values (e.g. from System properties).
app.name=MyApp
app.description=${app.name} is a Spring Boot application
dev:
    url: http://dev.bar.com
    name: Developer Setup
prod:
    url: http://foo.bar.com
    name: My Cool App

Placeholders and dynamic resolution

Using the @Value("${property}") annotation to inject configuration properties can sometimes be cumbersome, especially if you are working with multiple properties or your data is hierarchical in nature. Spring Boot provides an alternative method of working with properties that allows strongly typed beans to govern and validate the configuration of your application. For example:
@Component
@ConfigurationProperties(prefix="connection")
public class ConnectionSettings {

    private String username;

    private InetAddress remoteAddress;

    // ... getters and setters

}

Uses a Java based variant: 

@Configuration
// @Profile("production")
@EnableAutoConfiguration(exclude={EmbeddedDatabaseConfiguration.class})
public class MyConfiguration {
}

4. Apache Deltaspike

The Apache DeltaSpike configuration system enables providing a default configuration inside the binary and allowing to amend this configuration (e.g. database credentials, some URLs from remote REST or SOAP endpoints, etc) from outside like environment settings, JNDI or the current ProjectStage.

Drop-In Configuration

The mechanism also allows for dynamic configuration in case of a JAR drop-in. By adding some JAR to the classpath, all it's contained configuration will get picked up and considered in the property value evaluation. You could also use this mechanism to switch implementations of some SPI (Service Provider Interface) in your own code.

CDI-Extension Configuration

In some cases low-level configs are needed e.g. during the bootstrapping process of the CDI container.
The good news: our DeltaSpike configuration mechanism does not rely on any other EE mechanism to be booted. Which means it can perfectly get used to even configure those parts itself. Since the mechanism doesn't rely on CDI it can for example be used to configure CDI-Extensions.
Currently this is e.g. used to configure the value of the current ProjectStage, configured values which can be used in the expressions for @Exclude, 'Deactivatable', etc. DeltaSpike needs such a low-level approach for several features internally, but users can utilize it for their own needs as well. This is done by using the ConfigResolver which resolves and caches ConfigSources per application.

Userland Configuration

DeltaSpike also provides a mechanism to inject those configured values using the @ConfigProperty CDI Qualifier.

ConfigSources provided by default

Per default there are implementations for the following config sources (listed in the lookup order):
  • System properties (deltaspike_ordinal = 400)
  • Environment properties (deltaspike_ordinal = 300)
  • JNDI values (deltaspike_ordinal = 200, the base name is "java:comp/env/deltaspike/")
  • Properties file values (apache-deltaspike.properties) (deltaspike_ordinal = 100, default filename is "META-INF/apache-deltaspike.properties")
It's possible to change this order and to add custom config sources.
To add a custom config-source, you have to implement the interface ConfigSource and register your implementation in a file /META-INF/services/org.apache.deltaspike.core.spi.config.ConfigSource by writing the fully qualified class name of the custom implementation/s into it.

Type-safe configuration

Finally DeltaSpike provides also a way to directly inject configured values into your code via the qualifier @ConfigProperty.
@ApplicationScoped
public class SomeRandomService
{
    @Inject
    @ConfigProperty(name = "endpoint.poll.interval")
    private Integer pollInterval;

    @Inject
    @ConfigProperty(name = "endpoint.poll.servername")
    private String pollUrl;

    ...
 }

5. Apache Commons Configuration

Configuration parameters may be loaded from the following sources:
  • Properties files
  • XML documents
  • Windows INI files
  • Property list files (plist)
  • JNDI
  • JDBC Datasource
  • System properties
  • Applet parameters
  • Servlet parameters
Different configuration sources can be mixed using a ConfigurationFactory and a CompositeConfiguration. Additional sources of configuration parameters can be created by using custom configuration objects. This customization can be achieved by extendingAbstractConfiguration or AbstractFileConfiguration.
The full Javadoc API documentation is available here.
For manipulating properties or their values the following methods can be used:
addProperty()
Adds a new property to the configuration. If this property already exists, another value is added to it (so it becomes a multi-valued property).
clearProperty()
Removes the specified property from the configuration.
setProperty()
Overwrites the value of the specified property. This is the same as removing the property and then calling addProperty() with the new property value.
clear()
Wipes out the whole configuration

6. Mechanism provided with the JDK

System Properties

Of course the well known Java system properties are always an easy way to configure things. By adding additional properties on the command line starting the Java process additional properties can be defined, e.g.

java -Dfoor.bar=notYet  <mainClass>

will start the given <mainClass>. Hereby the value from foor.bar can be extracted by calling System.getProperty("foo.bar"); This mechanism is widely used, but also has its limitations, mainly because it is a global configuration mechanism (the value is the same for the whole VM). Also adding very large amounts of system properties may lead to comlpex installation and deployment scenarios.

java.util.Properties

Of course, you could use simple property files. E.g. you can add a property file foobar.properties to your classpath (or a file, if you like)and then read it using the corresponding reader methods:

Properties props = new Properties();
props.read(getClass().loadResource("foobar.properties");

Additionally the JDK also support storing properties using a XML format:

Properties props = new Properties();
props.readfromXML(getClass().loadResource("foobar.properties");

Unfortunately there is no support for overriding of properties. So this mechanism is very limited.

java.util.Preferences

Another API present in the JDK is java.util.preferences. It models a tree of nodes, where each node can have arbitrary attributes:

Prefereces systemPrefs = Preferences.systemRoot();
Prefereces userPrefs = Preferences.userRoot();

String configuredValue = systemPrefs.get("a.b.c.myKey", "myDefaultValue");
int configuredIntValue = userPrefs.get("my.foo.intValue", 190);

The key used above hereby resolve to paths in the tree, so "a.b.c.myKey", references the following node:

<root>
  \_ a
      \_ b
         \_ c

myKey is finally the key looked up in the node c.

Basically a node supports the following data types:
  • String
  • int
  • long
  • float
  • double
  • byte[]
So the API is simple but in practice has shown to be very limited and not appropriate as a base for modelling configuration, because
  • The API's absrtactions are modelled by abstract classes, which prevents the flexibility required.
  • Registering additional PreferencesFactory instances (SPI) may interfere with texisting code.
  • Access to the tree is widely synchronized, which is not acceptable for accessing configuration in a EE environment.
  • The Preferences SPI also supports writing configuration back, but is in combination with remote backends basically very cumbersome to implement and use.

7. JFig

See http://jfig.sourceforge.net/

JFig allows developers to:

  • Store application configuration in one common repository of XML files
  • Access configuration data using one common, convenient interface
  • Easily define multiple configurations, dynamically modifying those variables that need to change in different situations.
  • Eliminate the error prone practice of defining the same configuration variables in multiple locations
  • Ease the management, deployment, and control of configuration files
JFig structures configuration as a hierarchy of configuration files. It supports .xml and .ini files:

XML Format
<configuration>
<include name=”base.config.xml”/>
      <section name=”locos”>
<entry key=”instance” value=”development” />
      </section>
<section name=”Paths”>
<entry key=”locosHome” value=”d:/[locos]{instance}/project/” />
<entry key=”locosExternal” value=”d:/external/[locos]{instance}/” />
</section>

<section name=”attachments”>
<entry key=”attachmentDirectory” value=”[paths]{locosExternal}attachments/”/>
     </section>
</configuration>

INI File Format
Section names are in brackets, followed by key/value pairs.
Following is a sample file:
[project]
instance=development
timeout=5

[Paths]
locosHome=d:/project/[locos]{instance}/
locosExternal=d:/[locos]{instance}/

[attachments]         
attachmentDirectory=[paths]{locosHome}attachments/

Hereby JFig allows to read multiple files. The “include” directive instructs the JFig parser to find the included configuration file. Files are parsed in reverse order. Thus, if a prod config file includes a base config file, the base config is parsed first.


Variable Substitution

The system provides substitution and cross referencing of variables. Substitution variables are in the format: [section]{key}. The second value, section Paths, key locosHome, uses the value of project, instance to build its value. So, locosHome resolves to d:/project/development . The next entry, attachmentDirectory, uses the previous entry to build its value. Thus, attachmentDirectory resolves to d:/project/development/attachment.
Also JFig allows to reference system properties:
<entry key=”docomentDir” value=”$user.home$/documents” />

Accessing Configuration

Configuration then can be accessed as follows:

JFig.getInstance().getValue(sectionName, key) Or 
JFig. getInstance().getValue(sectionName, key, defaultValue)

If you use a default value and the section/key is not found, the default value will return. If you don’t use a default value and the section/key is not found, ConfigException is thrown.

To set JFig values, use 
JFig.setConfigurationValue(sectionName, key, newValue);

For the most part, however, config values are set during the initial parsing of one or more ini files.

To show a complete listing of JFig values, use 
JFig.getInstance().printConfigurationDictionary()

8. Carbon


See http://carbon.sourceforge.net/modules/core/docs/config/Usage.html
The Carbon configuration model takes two steps to improve the use of externalized data files for configuration. The first is to provide a simple configuration hierarchy based on the model of folders and documents. The second is to provide more advanced integration between the data and code using something called data binding. This provides much more advanced error checking as well as a consistent method for managing the data and most importantly strongly-typed access to the data.
The primary interface to configuration data is through extensions of the Configuration interface. The configuration service, through data binding, is able to implement any interface that extends the Configuration interface and adheres to the JavaBean™ Specification for properties.
package examples; 
public interface WebServiceConfiguration extends Configuration {
 String getServiceName(); void setServiceName(String value); String getServiceAddress(); void setServiceAddress(String serviceAddress);
 boolean isSecureConnection(); void setSecureConnection(boolean secureConnection);
 Class TransportClass = org.apache.soap.transport.SOAPHTTPConnection.class; Class getTransportClass(); void setTransportClass(Class value);
}
A Carbon configuration file then would look as follows:
<Configuration ConfigurationInterface="examples.WebServiceConfiguration">
   <ServiceName>HelloWorld</ServiceName>
   <ServiceAddress>http://ws.examples/HelloWorld</ServiceAddress>
   <SecureConnection>true</SecureConnection>
</Configuration>

The configuration can then be accessed as follows:

WebServiceConfiguration myConfig = (WebServiceConfiguration)
    Config.getInstance().fetchConfiguration("/WS/example"); 
if (myConfig.isSecure()) {
    // ... use ssl socket factory
    ...
else {
    // ... use standard socket factory 
    ... 
Transport transport = myConfig.getTransportClass().newInstance(); transport.send(new URL(myConfig.getServiceAddress()), "hello", // .....

Additionally there is also a weakly typed variant:

<Configuration ConfigurationInterface="org.sape.carbon.core.config.PropertyConfiguration">
 <MyInt>4</MyInt>
 <MyFloat>6.6</MyFloat> <MyString>Hello, World!</MyString>
 <MyClass>org.sape.carbon.core.config.format.test.ConfigurationFormatServiceTest
   </MyClass> 
 <Name> <First>John</First> <Last>Doe</Last> </Name>
</Configuration>
Which can be accessed as follows:


int num = myConfig.getIntProperty("MyInt");String firstname = myConfig.getProperty("Name.First");

9. Owner

See https://github.com/lviggiano/owner

The approach used by OWNER is to define a Java interface associated to a properties file.
Suppose your properties file is defined as ServerConfig.properties:
port=80
hostname=foobar.com
maxThreads=100
To access this property you need to define a convenient Java interface in ServerConfig.java:
public interface ServerConfig extends Config {
    int port();
    String hostname();
    int maxThreads();
}
OWNER calls this interface the Properties Mapping Interface or just Mapping Interface since its goal is to map Properties into a an easy to use piece of code. Finally, you can let OWNER read and assign your config as follows:
ServerConfig cfg = ConfigFactory.create(ServerConfig.class);
This looks rater simple and straight forward. But owner has a bunch of additional interesting features and is one of the most sophisticated solutions I have found so far

  • default values support
  • collection support, including custom tokenizers
  • type conversion support
  • overriding mechanisms
  • placeholders (called variable expansion) and dynamic resolution
  • mutability and configuration hot reload (both synchronous and asynchronous)
  • listening to reload events
  • explicit modelling of accessibility and mutability of configuration
  • Meta-Configuration

10. Other Configuration Mechanisms

I encountered several configuration frameworks and mechanism during my work life so far. Nevertheless I also want to describe one of the more powerful ones, just to show some of the features/requirements identified in bigger companies.
Typically such custom solution either use one or more of the mechanisms described above, or the companies decided to write a feasible configuration solution themselves.

Hereby it is noteable that
  • most of the solutions are based on simple key, value pairs, with String values only.
  • some companies added also support for different Collection types of String, but in general this concepts has shown to add more complexity and issues than it provides advantages.
  • some of them use flat keys, but most of them define a tree structure, similar to what the JDK's preferences API is doing.
  • the system differentiate between the execution context, such as
    • executed during system startup (system class loader)
    • executed during ear startup/shutdown (ear class loader)
    • executed during application startup (application class loader)
    • executed during a request, e.g. a http request, or an EJB called.
    • also a stage is defined, which may look very different for different companies
  • the system define different override rules, e.g.
    • system properties override everythging
    • application properties override ear config
    • ear config overrides system config
  • some of them also map environment,system properties and the command line arguments into the global configuration tree.
  • some also use additional environment settings, such as tier, network zone, security policy, server or JVM/app server instance names to define further possibilities for customizing the final configuration active.


23 comments:

  1. I suggest you to take a look at http://owner.aeonbits.org/
    It is a great configuration framework, based on the idea of type-safety (you call a method instead of using directly strings).
    It is based on plain old properties, and does not use annotations (because it aims to be type-safe). Other than that it may be an inspiration for you.

    ReplyDelete
  2. I have added a small section about OWNER, tried to create some appetite for it, since it looks as one of the very advanced solutions around there...

    ReplyDelete
  3. Hello Anatole,

    Would you consider adding CFG4J (http://cfg4j.org) to your comparison? It's a configuration library for Java distributed apps - as OWNER it offers type-safety and ease of use but supports many more backends (e.g. Consul, Git repos) and has some distributed environment-friendly features like fallback strategies, re-tries and caching.

    ReplyDelete
  4. Under the properties section the statment "Unfortunately there is no support for overriding of properties. So this mechanism is very limited." Is not strictly correct if the properties are created with the constructor new Properties(Properties defaults). Any new properties added override the default values supplied to the constructed properties, if a key is removed from constructed properties the value from the defaults is returned. Similarly if a key does not exist in the properties an attempt is made to read the value from the specified defaults. Calls to the constructor with defaults are not limited to a single depth and can be chained.

    ReplyDelete
  5. I have read your blog its very attractive and impressive. I like it your blog.

    Java Online Training Java EE Online Training Java EE Online Training Java 8 online training Java 8 online training

    Java Online Training from India Java Online Training from India Core Java Training Online Core Java Training Online Java Training InstitutesJava Training Institutes

    ReplyDelete
  6. I have read your blog its very attractive and impressive. I like it your blog.

    Java Online Training Java EE Online Training Java EE Online Training Java 8 online training Java 8 online training

    Java Online Training from India Java Online Training from India Core Java Training Online Core Java Training Online Java Training InstitutesJava Training Institutes

    ReplyDelete
  7. Thank You for sharing your article. I like it. We provide TIBCO Online Training in Hyderabad.

    ReplyDelete
  8. The official CORE Java API, contained in the Android (Google), SE (OpenJDK and Oracle), MicroEJ (IS2T). These packages (java.* packages) are the core Java language packages, meaning that programmers using the Java language had to use them in order to make any worthwhile use of the java language.
    optional APIs that can be downloaded separately. The specification of these APIs are defined according to many different organizations in the world (Alljoyn, OSGi, Eclipse, JCP, E-S-R, ... ).

    AngularJS Certification Training in Chennai

    AngularJS Training Institute in Chennai ==> Core Java Training in Chennai

    ReplyDelete
  9. perfect explanation about java programming .its very useful.thanks for your valuable information. java training in chennai | java training in velachery

    ReplyDelete
  10. Thanks a lot! You made a new blog entry to answer my question; I really appreciate your time and effort.
    java training center in chennai |
    best java training in chennai

    ReplyDelete
  11. I accept there are numerous more pleasurable open doors ahead for people that took a gander at your site.we are providing ReactJs training in Chennai.
    For more details: ReactJs training in Velachery | ReactJs training in chennai

    ReplyDelete
  12. Great Article… I love to read your articles because your writing style is too good,
    its is very very helpful for all of us and I never get bored while reading your article because,
    they are becomes a more and more interesting from the starting lines until the end.
    Java training in Annanagar
    Java training in Chennai
    Java training in Chennai
    Java training in Electronic city
    Java training in Marathahalli

    ReplyDelete
  13. You can join the free orientation at Workday training institutes in Bangalore and they located at Workday training institutes in delhi. It is highly recommendable for you to enroll at Workday training institutes in UK and you can have a great market for jobs in USA so join this wonderful Workday training institutes in USA and in India this Workday training institutes in Noida is really best institute.

    ReplyDelete
  14. I have recently found an excellent Salesforce Training in India whose faculty is exceptional and you can also try this Salesforce Training and Certification in Jaipur whose syllabus is state of art. Here at Salesforce Training in Mumbai instructors are perfect to teach salesforce crm. My advice for you is to join demo at Salesforce training in Pune | Course Cost and in weekends try this best Salesforce Training in Noida | Course Cost who is providing great teaching services on Salesforce Training in Delhi and Fee Details.

    ReplyDelete
  15. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    workday online training
    best workday online training
    top workday online training

    ReplyDelete
  16. Crypto-currency as a modern form of the digital asset has received a worldwide acclaim for easy and faster financial transactions and its awareness among people have allowed them to take more interest in the field thus opening up new and advanced ways of making payments. Crypto.com Referral Code with the growing demand of this global phenomenon more,new traders and business owners are now willing to invest in this currency platform despite its fluctuating prices however it is quite difficult to choose the best one when the market is full. In the list of crypto-currencies bit-coins is one of the oldest and more popular Crypto.com Referral Code for the last few years. It is basically used for trading goods and services and has become the part of the so-called computerized block-chain system allowing anyone to use it thus increasing the craze among the public, Crypto.com Referral Code.

    Common people who are willing to purchase BTC can use an online wallet system for buying them safely in exchange of cash or credit cards and in a comfortable way from the thousands of BTC foundations around the world and keep them as assets for the future. Due to its popularity, many corporate investors are now accepting them as cross-border payments and the rise is unstoppable. With the advent of the internet and mobile devices,information gathering has become quite easy as a result the BTC financial transactions are accessible and its price is set in accordance with people’s choice and preferences thus leading to a profitable investment with Crypto.com Referral Code Code. Recent surveys have also proved that instability is good for BTC exchange as if there is instability and political unrest in the country due to which banks suffer then investing in BTC can surely be a better option. Again bit-coin transaction fees are pretty cheaper and a more convenient technology for making contracts thus attracting the crowd. The BTC can also be converted into different fiat currencies and is used for trading of securities, for land titles, document stamping, public rewards and vice versa.

    Another advanced block-chain project is Ethereumor the ETH which has served much more than just a digital form of crypto-currency Crypto.com Referral Code and its popularity in the last few decades have allowed billions of people to hold wallets for them. With the ease of the online world,the ETH have allowed the retailers and business organizations to accept them for trading purposes, therefore, can serve as the future of the financial system.

    ReplyDelete
  17. Our full Lace Front Wigs are all hand made with a lace cap. They are manufactured with thin lace sewn on top of the cap. Individual hairs are then sewn onto the thin lace. Each lace wig has lace all around the unit which will need to be cut prior to securing the wig to your head. You will need to cut along the hairline around your entire head. By doing so, you will be able to wear your hair anyway you like. You can even style ponytails, up-dos, etc. Once the Lace Wigs is successfully applied, it will appear that all the hair is growing directly from your head!

    Lace front wigs are hand-made with lace front cap & machine weft at back. Lace front wigs are manufactured with a thin lace that extends from ear to ear across the hairline. When you receive the wig, the lace will be quite long in the front. Cut and style according to your preference, as you will need to apply adhesive along the front of the wig. Once the wig is applied, you will still have Lace Wigs with a very natural appearance.
    TeamWigz Provide the Best Lace Front Wigs and Lace Wigs in Johannesburg and South Africa.

    ReplyDelete
  18. If you live in Los Angeles and need Washer Repair Service Los Angeles, we are the company to call. We have years of experience repairing all makes and models of washer. Our service is fast, effective and affordable. Plus, we stand behind our work. If you want to make sure your washer is functioning at optimal efficiency, call us. We will be there in a flash and can quickly solve any problem you are having with your washer. Whether you have an older model washer or one of the latest, high-tech ones, we can get it working in a flash. We have long provided the washer repair Los Angeles residents need.

    No matter what type of washer problem you are having, we can help. We can fix your washer if it won’t start, stopped spinning, won’t stop, is making strange noises, tripping the breaker, the drum isn’t turning or any other issue.

    We have a team of well-trained, highly skilled technicians with many years of experience repairing Washer Repairs in Los Angeles of all types. Our technicians are licensed, bonded and insured. Plus they are prompt, courteous, friendly and professional. They have the tools and expertise necessary to identify and repair any problem you are having with your washer. They will let you know if the problem is a part that is worn or broken and if the part needs to be repaired or replaced. They will let you know what the job will cost in advance, so you will not have any surprises when the work is done.

    We charge some of the lowest prices in the industry. We do a volume business, so we buy our replacement parts in bulk and pass the savings on to you. Plus, we only use quality replacement parts, so you can rest assured you are getting great value for your money. We will let you know if it will cost less and be just as sturdy if we repair or replace any part that is worn or broken. That gives you the power to decide what it will cost to get your washer working and keep it working for a long time to come.

    ReplyDelete
  19. If you live in Los Angeles and need Refrigerator Repair Service Los Angeles, we are the company to call. We have years of experience repairing all makes and models of refrigerator. Our service is fast, effective and affordable. Plus, we stand behind our work. If you want to make sure your refrigerator is functioning at optimal efficiency, call us. We will be there in a flash and can quickly solve any problem you are having with your refrigerator. Whether you have an older model refrigerator or one of the latest, high-tech ones, we can get it working in a flash. We have long provided the refrigerator repair Los Angeles residents need.

    No matter what type of refrigerator repair problem you are having, we can help. We can fix your refrigerator if it won’t start, stopped cooling, won’t stop, is making strange noises, tripping the breaker, the compressor isn’t turning or any other issue.

    We have a team of well-trained, highly skilled technicians with many years of experience repairing refrigerator of all types. Our technicians are licensed, bonded and insured. Plus they are prompt, courteous, friendly and professional. They have the tools and expertise necessary to identify and repair any problem you are having with your refrigerator. They will let you know if the problem is a part that is worn or broken and if the part needs to be repaired or replaced. They will let you know what the job will cost in advance, so you will not have any surprises when the work is done.

    We charge some of the lowest prices for Refrigerator Repair Los Angeles in the industry. We do a volume business, so we buy our replacement parts in bulk and pass the savings on to you. Plus, we only use quality replacement parts, so you can rest assured you are getting great value for your money. We will let you know if it will cost less and be just as sturdy if we repair or replace any part that is worn or broken. That gives you the power to decide what it will cost to get your refrigerator working and keep it working for a long time to come.

    ReplyDelete